Question: . "github.com/onsi/gomega" ) <ins> var serveHostnameImage string = "gcr.io/google_containers/serve_hostname:1.1" </ins> func resizeNodeInstanceGroup(size int) error { // TODO: make this hit the compute API directly instread of shelling out to gcloud. output, err := exec.Command("gcloud", "preview", "managed-instance-groups", "--project="+testContext.CloudConfig.ProjectID, "--zone="+testContext.CloudConfig.Zone,

Master tracking bug:
The scenario of the test is the following: a service an RC with as many pods as the number of nodes for pods created, running and responding the first node and get the pod from it ... It is assumed that the first node has a pod - that's because kubernetes is supposed to distribute pods for which there is a service evenly among nodes. So we expect that each node has one pod. This turns out to be false. (in 80% cases the first node doesn't have a pod.
Thanks . cc: Is it a valid assumption that pods should be evenly spread across nodes? I assume not? If even spreading is a valid assumption, then there's a bug in the system. If not, then we need to fix this test.
To clarify the above, it's perfectly sensible to me that k8s should in an ideal world spread pods that constitute a service across as many nodes as possible. But I doubt whether our scheduler has such logic yet?
Our spreading is only a probabilistic guarantee. So this is incorrect test behavior.
We do actually have spreading logic. To determine if this is a scheduler bug, you'd have to know the state of the system more exactly. Spreading is balanced with resource usage, for example-- if something big was a little slow leaving the node, it could cause it to not get a pod. Or if it was briefly unavailable.
OK, I'd suggest that we change the test then, to not assume that pods are perfectly evenly spread across nodes.
, is there a way to run the scheduler with debug logs? I'm interested about the scores for nodes. (even if it won't help me in this issue, it's a nice exercise) I suppose the test fails because CPU utilization has a significant impact on scheduling pods - more that having a service. That's why all pods are scheduled on the same node. I'm going to remove the requirement that there is one pod per node (I'll still create a service though - just to make pods spread more evenly). I'll simply remove from the network one node that has a pod (without assuming that the first one has). I believe the scenario will still test everything we need ( please correct me if I'm wrong). The PR should be ready today.

By the way, a side effect of this test is that in a 2-node cluster, all add-ons will end up on one node, occupying 0.9 cores - this setup will stay for further tests, so it's good to be aware of it. I'll write this comment in the test code as well.
is ready for review. Possible second round will be needed. No changes in production code have been made (tests only).
I think if you set v=10 you'll see the log messages with that info in them.
This scenario is OK.
, sorry for such simple questions, but how do I set v=10?
you need to edit and change to in the command line args. The kubelet should then restart the container (validate with ) if it doesn't, should fix it.
Status: waiting for review of


Question: else # no file exists. Adding a new file <del> cat <<EOF > "$juDIR/junit-$suite.xml" </del> <ins> cat <<EOF > "$juDIR/junit_$suite.xml" </ins> <?xml version="1.0" encoding="UTF-8"?> <testsuites> <testsuite failures="$errors" assertions="$assertions" name="$suite" tests="1" errors="$errors" time="$total">

is not picking the cmd test junit output due to file naming issue. The file name should match regex /assign
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig testing


Question: if new, ok := newContainerIndex[old.Name]; !ok { allErrs = append(allErrs, field.Forbidden(specPath, fmt.Sprintf("existing ephemeral containers %q may not be removedn", old.Name))) } else if !apiequality.Semantic.DeepEqual(old, *new) { <del> specDiff := diff.ObjectDiff(old, *new) </del> <ins> specDiff := cmp.Diff(old, *new) </ins> allErrs = append(allErrs, field.Forbidden(specPath, fmt.Sprintf("existing ephemeral containers %q may not be changedn%v", old.Name, specDiff))) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --pod with nodeSeletor edit pod, delete the nodeSeletor field report error with: ! The errors message should displayed as: Because the nodeSeletor is been deleted with this edit. Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/remove-sig api-machinery /sig apps
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle stale
/remove-lifecycle rotten


Question: // Update server with new CRD condition. _, err = ec.crdClient.CustomResourceDefinitions().UpdateStatus(crd) <ins> if apierrors.IsNotFound(err) || apierrors.IsConflict(err) { // deleted or changed in the meantime, we'll get called again return nil } </ins> if err != nil { return err }

k8s version: v1.9.3 When I create customresourcedefinition, There is error like Operation cannot be fulfilled on "": the object has been modified; please apply your changes to the latest version and try again . apiVersion: kind: CustomResourceDefinition metadata: name: annotations: "": crd-install labels: app: istio-pilot spec: group: names: kind: VirtualService listKind: VirtualServiceList plural: virtualservices singular: virtualservice categories: istio-io networking-istio-io scope: Namespaced version: v1alpha3 So what is the reason of it?
a few questions: where are you seeing the error, in the apiserver log or in the output of the kubectl command? what kubectl command are you using to create the object? are you running HA apiservers?
The log is shown by apiserver. I run k8s in HA apiservers. The command I used is "kubectl apply -f "+file.
API servers attempt to update CRD status to indicate whether the names were available. In an HA apiserver setup, the API servers race to do so, and all but the first lose the race with a resource conflict error. That is not unexpected. Opened to avoid logging normal API responses in that scenario.
Thanks ! But I want to know whether the CRD has been set successfully in HA apiservers ? I use the command " kubectl get crd -o yaml" and get the following results. ! I see its status is true. But when i create a virtualservices, it returns error. Error from server (Timeout): error when creating "cms-api-": Timeout: request did not complete within allowed duration ! So i want to know how to solve this error.
/assign
Excuse me. Do you have some greate ideas to solve this issue??
There is a PR for this sent by
I see your request is timed-out not failed. In HA, I expect the request to fail in the first seconds of the CRD creation, but after that it should go through. I need more logs during this timeout or a consistent way to reproduce it.


Question: } // Pass all parameters as volume labels for Portworx server-side processing <del> if len(p.options.Parameters) > 0 { spec.VolumeLabels = p.options.Parameters } else { </del> <ins> if spec.VolumeLabels == nil { </ins> spec.VolumeLabels = make(map[string]string, 0) } <ins> for k, v := range p.options.Parameters { spec.VolumeLabels[k] = v } </ins> // Update the requested size in the spec spec.Size = uint64(requestGiB * volumehelpers.GiB)

What happened: The Portworx volume driver is accessing the storage parameters and also modifying it. This is a shared map object used by all PVCs being created at that time from the same StorageClass. This causes 2 issues: Portworx driver ends up saving annotations of one PVC into the parameters of another PVC as both CreateVolume calls use the same map. See controller manager crashes as 2 CreateVolume routines try to modify the same map. This is a big problem since the controller manager container crashes half way into creating a volume resulting in the volume being present in the backend portworx cluster but not bound in Kubernetes. What you expected to happen: CreateVolume call for one PVC should be completely isolated from another PVC How to reproduce it (as minimally and precisely as possible): Create a large number of PVCs using the Portworx volume driver using the same StorageClass. Anything else we need to know?: Environment: n/a Kubernetes version (use ): 1.13.2 Cloud provider or hardware configuration: n/a OS (e.g: ): n/a Kernel (e.g. ): n/a Install tools: Kubernetes cluster with Portworx installed Others:
/sig storage
I'll take care of fixing this


Question: // Addf logs info immediately. func (passthroughLogger) Addf(format string, data ...interface{}) { <del> glog.Infof(format, data...) </del> <ins> glog.InfoDepth(1, fmt.Sprintf(format, data...)) </ins> } // DefaultStacktracePred is the default implementation of StacktracePred.

the file and line for these log statements is in the That's not helpful I1031 11:09:06. ] 'foo' has no storage object I1031 11:09:06. ] 'pods' is not a redirector I'd rather see Maybe use runtime.Caller in httplog to somehow fake the log to come from the caller's file/line.
Yeah-- actually this should be a feature of the glog package. Would be awesome to do:
Followup: glog is an opensource copy of a google-internal library; there's already something enabling this internally; we'll try to get a new version pushed externally in the next week or so. Then we can fix this.
Awesome. On Oct 31, 2014 2:53 PM, "Daniel Smith" wrote:
should meet your requirements.
+1, I would love to have this.


Question: }, } r := &Reflector{ <del> name:                   "test-reflector", listerWatcher:          lw, store:                  NewFIFO(MetaNamespaceKeyFunc), initConnBackoffManager: bm, clock:                  fakeClock, watchErrorHandler:      WatchErrorHandler(DefaultWatchErrorHandler), </del> <ins> name:              "test-reflector", listerWatcher:     lw, store:             NewFIFO(MetaNamespaceKeyFunc), backoffManager:    bm, clock:             fakeClock, watchErrorHandler: WatchErrorHandler(DefaultWatchErrorHandler), </ins> } start := fakeClock.Now() err := r.ListAndWatch(stopCh)

Currently, reflector uses two backoff managers: One is used for initial list call and other for watch call. Imagine following scenaraion when k8s apiserver is under very heavy load: the beginning, reflector spams k8s apiserver (with exponential backoff) with calls until one of them succeeds (due to APF throtling and receiving ) tries to run request and uses separate backoffManager - so again spams k8s apiserver with requests Watch does not succeed during Etcd compaction window (2.5 - 5 minutes usually) at some point it will receive and go back to point 1, spamming k8s apiserver again with List calls (because first backoffManager for List got reseted after 2 minutes of retries for Watch requests) I propose we combine these two backoff managers into one. If we combine these two backoffManagers into one, it will work like this: At the beginning, reflector spams k8s apiserver (still with exponential backoff) with calls until one of them succeeds Tries to make first watch call instantly (as there is no backoff on this path) If it fails, it will makre retries every (30, 60)s If watch request fails all the times during Etcd compaction window it will go back to List call, but already being throttled with (30, 60)s window. /sig scalability
/cc
/assign
I think the proposed solution makes sense to me. Probably before we start, we should to understand why we ended up with two back off managers by looking into PRs touching them. Maybe the code authors had some specific case in mind.
for thoughts too I agree it makes sense, but I don't know the historical context here.
/sig api-machinery
/triage accepted


Question: } pvLister = append(pvLister, pv) } } return pvLister }

handles cases when csiNode is nil. This is condition is used by cluster-autoscaler internal working. However, the log line refers to causing seg fault
/sig scheduling /sig storage
/sig storage /priority important-soon /triage accepted
/sig storage /sig scheduling Note that the lister returns an error for the case when the doesn't exist. But it's ignored here: It looks like the log line that causes the panic was introduced in in 1.25+
cc
/assign
As far as I can tell this issue is only fixed in version 1.27 (by ). Is there a plan to port it back to releases 1.25 and 1.26 as well?
it has already been picked to 1.25 and 1.26 () and will be in the next patch releases for those versions
Great. Thank you for the pointer.


Question: if hosts.Len() > 1 { return } <del> time.Sleep(time.Duration(svcSessionAffinityTimeout) * time.Second) </del> <ins> // In some case, ipvs didn't deleted the persistent connection after timeout expired, // use 'ipvsadm -lnc' command can found the expire time become '13171233:02' after '00:00' // // pro expire state       source             virtual            destination // TCP 00:00  NONE        10.105.253.160:0   10.105.253.160:80  10.244.1.25:9376 // // pro expire state       source             virtual            destination // TCP 13171233:02 NONE        10.105.253.160:0   10.105.253.160:80  10.244.1.25:9376 // // And 2 seconds later, the connection will be ensure deleted, // so we sleep 'svcSessionAffinityTimeout+5' seconds to avoid this issue. // TODO: figure out why the expired connection didn't be deleted and fix this issue in ipvs side. time.Sleep(time.Duration(svcSessionAffinityTimeout+5) * time.Second) </ins> } } framework.Fail("Session is sticky after reaching the timeout")

Which jobs are flaking: https://k8s- Which test(s) are flaking: Services should have session affinity timeout work for service with type clusterIP Services should have session affinity timeout work for NodePort service We have other flaky cases, like , that should also bu resolved. But cases also run in , so I suggest we solve this problem first. Testgrid link: https://k8s- Reason for failure: In ipvs mode, we set session affinity timeout as 125s, the test case is: run host command to send a request, mark returned hostName 125s another request, if return different hostName, test succed. I think the reason is, in the second request, ipvs connection may still in sessionAffinity, send another request will result in seffinity restart count, eventually result in this test case flaky.
/sig network /area ipvs
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign /remove-triage unresolved
This keeps failing https://k8s-


Question: "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" <ins> utilrand "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" "k8s.io/client-go/dynamic" clientset "k8s.io/client-go/kubernetes" appsclient "k8s.io/client-go/kubernetes/typed/apps/v1" watchtools "k8s.io/client-go/tools/watch" <ins> "k8s.io/client-go/util/retry" </ins> appsinternal "k8s.io/kubernetes/pkg/apis/apps" deploymentutil "k8s.io/kubernetes/pkg/controller/deployment/util" "k8s.io/kubernetes/test/e2e/framework"

[X] Test PR: kuberenetes/kubernetes [x] Two weeks soak end date: 1 July 2021 [x] Test promotion PR: kubernetes/kubernetes According to this APIsnoop query, there are still some remaining StatefulSet endpoints which are untested. Create a watch to track deployment events. Create a deployment with a static label. Confirm that the pods are running. Get the deployment status. Parse the response and confirm that the deployment status conditions can be listed. Update the deployment status. Confirm via the watch that the status has been updated. Patch the deployment status. Confirm via the watch that the status has been patched. Using an existing as a template for a new for deployment lifecycle test. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 1 point This test is also created with the goal of conformance promotion. /sig apps /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return job, nil } <del> // Return Unix Epoch Time </del> <ins> // getTimeHash returns Unix Epoch Time </ins> func getTimeHash(scheduledTime time.Time) int64 { return scheduledTime.Unix() }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: There are some unused commet, these are meaningless. And some function‘s comment does not follow go style. What you expected to happen: We should delete them all. and fix comment following golang style How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
can you edit the PR description in to say instead of ? That would automatically close this issue once the PR is merged. :)
OK, no problem~:)


Question: }, }, }, false}, <ins> {api.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "update-provider-id-when-not-set", }, }, api.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "update-provider-id-when-not-set", }, Spec: api.NodeSpec{ ProviderID: "provider:///new", }, }, true}, {api.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "update-provider-id-when-set", }, Spec: api.NodeSpec{ ProviderID: "provider:///old", }, }, api.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "update-provider-id-when-set", }, Spec: api.NodeSpec{ ProviderID: "provider:///new", }, }, false}, </ins> } for i, test := range tests { test.oldNode.ObjectMeta.ResourceVersion = "1"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Follow up for , With the change for cloud-controller-manager updating providerID, the is to be updated. But it will fail later in node update call: Should also check validation rule: By adding following line will do the trick. But this will also allow users to manually edit node's providerID Shall we enforce that only cloud-controller-manager can update the providerID? What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig cluster-lifecycle
cc
/area cloudprovider
being able to update seems pretty reasonable to me. Can you get a PR in for that since we're close to code freeze?
cc
I found this issue when validating against a cluster running azure cloud provider, could you please also check whether it is the same case from you side? serves somewhat like an identity for nodes, thus it wouldn't expect arbitrarily updating. A compromise proposal is to allow updates when the value is not set. This gives controller manager a chance to set it during node register. Created as a fix proposal.
thanks for the patch, I think I'll have some time this weekend to verify this fix!
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/cluster-lifecycle: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details


Question: # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips tallclair </del> <ins> andrewsykim danwinship dcbw lachie83 thockin </ins>

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the kube-proxy repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
Can I list you folks? Anyone else who wants to be in?
count on me if you need more people


Question: InitContainers: []v1.Container{ { SecurityContext: &v1.SecurityContext{ <del> RunAsUser: utilptr.Int64Ptr(1000), </del> <ins> RunAsUser: ptr.To[int64](1000), </ins> }, }, }, Containers: []v1.Container{ { SecurityContext: &v1.SecurityContext{ <del> RunAsUser: utilptr.Int64Ptr(1000), </del> <ins> RunAsUser: ptr.To[int64](1000), </ins> }, }, { SecurityContext: &v1.SecurityContext{ <del> RunAsUser: utilptr.Int64Ptr(1000), </del> <ins> RunAsUser: ptr.To[int64](1000), }, }, }, EphemeralContainers: []v1.EphemeralContainer{ { EphemeralContainerCommon: v1.EphemeralContainerCommon{ SecurityContext: &v1.SecurityContext{ RunAsUser: ptr.To[int64](1000), }, </ins> }, }, }, }, }, <del> wantFsUser: utilptr.Int64Ptr(1000), </del> <ins> wantFsUser: ptr.To[int64](1000), </ins> }, }

projected volumes exclude ephemeral containers in calculating fs user because "SecurityContext is not allowed" : This is no longer technically correct -- ephemeral containers are allowed to specify security context -- but it's a no-op because is guaranteed to be empty at pod creation when fs user is calculated. Still, the comment is wrong so let's clean it up to avoid confusion. /kind cleanup
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /assign
/priority backlog
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten


Question: } } <del> func attachVolumeToNode(volumeName, nodeName string) { </del> <ins> func attachVolumeToNode(nodes *v1.NodeList, volumeName, nodeName string) { </ins> // if nodeName exists, get the object.. if not create node object var node *v1.Node found := false

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --
/sig storage /triage accepted
Pr continue failed beacuse this data race, and i did an fix in ,it's ok now.
/assign
another instance
another instance: (PR: )
flaked again since was merged
fixed in
/reopen Can you backport to 1.21?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
opened
Hit this bug also while working on a fix for 1.19 - so maybe we should open another issue/pr for 1.20 and 1.19 ? At least 1.19 will be still around for quite a while
I think that would probably be approved. if anyone wants to take a stab at it
Thanks I'll check over weekend if i can provide this fix also to 1.20 and 1.19
Opened for 1.19 Opened for 1.20
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: apps/v1/Deployment apps/v1/ReplicaSet apps/v1/StatefulSet <del> extensions/v1beta1/Ingress </del> <ins> networking.k8s.io/v1/Ingress </ins> ) else read -ra KUBECTL_PRUNE_WHITELIST <<< "${KUBECTL_PRUNE_WHITELIST_OVERRIDE}"

What happened: Sweeping uses of extensions/v1beta1 in preparation for , the addon manager stood out as using extensions/v1beta1 Ingress by default: What you expected to happen: All components shipped in 1.21 must use Ingress in order to work properly against a 1.22 API server. /sig networking cc per cluster/addons/addon-manager/OWNERS cc I'm not sure what the addon manager does if given a prune list of Ingress objects while still using a manifest containing extensions/v1 Ingress objects... this needs to be tested. /milestone v1.21 /priority important-soon /lifecycle frozen
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: cc /priority important-soon /lifecycle frozen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage accepted
how "urgent" is this? Can wait until next tuesday? Otherwise I can work on the tests this weekend :)
important, not urgent. it should happen early in the 1.21 cycle, but does not require work out of business hours
Hello! I am from the Bug Triage team! This issue has not been updated for a few weeks, so I'd like to check on the status. Code freeze is starting March 9th, 2021 (about 1 week from now). We want to ensure that each PR has a chance to be merged before code freeze. As the Issue is tagged for 1.21, is it still planned for this release?
we're waiting to the PR to get merged :) I guess it's important for the v1.21 release, for the sake of extensions apiGroup deprecation :) Will ping reviewers on the PR


Question: domain:           domain, cache:            NewTreeCache(), cacheLock:        sync.RWMutex{}, <ins> nodesStore:       kcache.NewStore(kcache.MetaNamespaceKeyFunc), </ins> reverseRecordMap: make(map[string]*skymsg.Service), domainPath:       reverseArray(strings.Split(strings.TrimRight(domain, "."), ".")), federations:      federations,

Run like so:
nodeStore doesnt seem to be set anywhere (except in tests). Thats why code panics when we access it:
cc
Sent to fix this
thx


bug (): Originally posted by in cc /sig network /priority critical-urgent
fix in
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: // FindContainersByPod returns the containers that belong to the pod. func (c DockerContainers) FindContainersByPod(podUID types.UID, podFullName string) DockerContainers { containers := make(DockerContainers) for _, dockerContainer := range c { if len(dockerContainer.Names) == 0 { continue

Observed in testing somehting unrelated - prio We need a test that captures this - we can't let this slip through, Maybe same as ?
I just reproduced this with newly built cluster from HEAD. This is definitely a regression introduced lately. Will look into it in detail.


Question: job.Status.CompletedIndexes = succeededIndexes.String() } job.Status.UncountedTerminatedPods = nil <del> jm.enactJobFinished(&job, finishedCondition) </del> <ins> jobFinished := jm.enactJobFinished(&job, finishedCondition) </ins> if _, err := jm.updateStatusHandler(ctx, &job); err != nil { return forget, err } <ins> if jobFinished { jm.recordJobFinished(&job, finishedCondition) } </ins> if jobHasNewFailure && !IsJobFinished(&job) { // returning an error will re-enqueue Job after the backoff period

Occasionally job completions are counted twice (or potentially multiple times?) in the metric. This happens for about 20% of jobs, but may depend on the Job specificity, which results in an overestimation of the number of finished jobs. The metric reflects the actual number of completed jobs. a kind cluster with prometheus monitoring stack (for example follow this: ). the number of completed jobs by inspecting the metric and delete a job multiple times in a loop Example job: Example repro script: . Check the number of completed jobs by inspecting the metric and the job Yes, in the kube-controller-logs we find the following log lines which are related and probably indicate repeated updated which results in double counting. <details</details<detailsReproducible with kind </details<details</details<details</details<details</details<details</details
/assign
FYI
/sig apps
/triage accepted


Question: "syscall" "time" <del> "k8s.io/apimachinery/pkg/apis/meta/v1" </del> <ins> v1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> statsapi "k8s.io/kubelet/pkg/apis/stats/v1alpha1" )

PIDPressure is not set to true when the system has insufficient pid resource. PIDPressure is set to true correctly when the system is insufficient in pid resource. See cause below The max pid is not solely controlled by which is referred by . But it is also controlled by (whichever is smaller). However, they have different semantics resulting in the same constraint to how many processes/threads can be running on the system. <details</details<details</details<details</details<details</details<details</details<details</details
/sig node
I asked the same issue before,
My opinion: the minimum of and is the right number to count, for Linux. That will provide better instrumentation. If you're reserving a number of task IDs (PIDs) for system use, that minimum is the number that you're reserving from.
/triage accepted /priority important-longterm
/retitle System maximum PID inconsiderate of sysctl 'threads-max'
BTW I made a thing that might help folks test a fix: Originally I used it to create PIDs, to test whether apps cope with long PIDs. This version keeps the threads around so that you see them in .


Question: // Total number of available pods (ready for at least minReadySeconds) targeted by this statefulset. // This is a beta field and requires enabling StatefulSetMinReadySeconds feature gate. <ins> // +optional </ins> AvailableReplicas int32 }

Looking at the for , it defines as a required attribute. But looking at the of , it states . Should this in fact be a required attribute? I come about this from using , a python package that generates models from the swagger definitions and thus builds a that requires an attribute that is not enabled. That only truly required attributes be required in the swagger definition Reproduction is not applicable as this is an issue with a spec No response Appeared in 1.22 (I believe) and affects current versions. Does not affect 1.21 n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
alpha: has that is planned for removal in beta. ( results in fields as non-required in the OpenAPI) beta: removes the this might be a question whether feature gate controlled fields must remain until the feature gate is removed (and how would validation work in that case). ~old feature gates graduations to beta enabled a feature by default.~ a topic related to ? but it does not apply to feature gates. /kind documentation /sig apps
cc (since you are the author)
new fields to existing types must always be marked as +optional, since clients are not required to specify them
please open a PR adding back the +optional tag, regenerate openapi, and open picks back to 1.23 and 1.22


Question: // localhostNodePorts indicates whether we allow NodePort services to be accessed // via localhost. localhostNodePorts bool <ins> // conntrackTCPLiberal indicates whether the system sets the kernel nf_conntrack_tcp_be_liberal conntrackTCPLiberal bool </ins> // nodePortAddresses selects the interfaces where nodePort works. nodePortAddresses *proxyutil.NodePortAddresses // networkInterfacer defines an interface for several net library functions.

When packets with sequence number out-of-window arrived k8s node, conntrack marked them as INVALID. kube-proxy will ignore them, without rewriting DNAT. However, there is no corresponding session link on the host, and the host sends a reset packet, causing the session link to be interrupted connection not reset This problem can be solved by command iptables -t filter -I INPUT -p tcp -m conntrack --ctstate INVALID -j DROP Similar But this issue "drop" is placed on the forward chain, our scenario needs to be placed on the input chain. $ kubectl version Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-04T17:57:25Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} <details</details<details</details<details</details<details</details<details</details
/sig network is this about pods with hostNetwork: true ? if not, why is this a kube-proxy issue if the problem is in INPUT?
/assign
Yeah, to be clear - filter INPUT applies to packets which are going to be delivered to a process. Is that kube-proxy's job to define?
The "How can we reproduce" sounds a bit like this always happens. Here is a better instruction
IMO is the best description of the problem:
This may be a better solution than let add rules in the INPUT chain
hehe, ironically we should decide to or , one is a sysctl call the other is one (or two) iptables rules. The iptables rule has also the side effect of breaking people (cc ) If we switch to set the option, the change does not seem to be backportable. I tend to agree with Lars, remove the iptables INVALID drop rule and set the sysctl, we are already setting more host sysctls what do you think?

it will be nice to have some iptables hackers opinion too
I tried to recreate this problem on virtual en (kvm/qemu) by requsting a 100MB file with curl and do; to provoke some packet loss. The transfer took longer time, but never failed.
IMO kube-proxy should not set any sysctls that are not literally required for functionality that the user has explicitly opted into (eg for most network plugins). Kube-proxy does not own the host network namespace, and it should not be doing things that will affect other people's host-network traffic, because if we do it's going to break some users. (See also: .) We can document that we think it's a good idea for users/distros to set , and if we think it's almost always a better idea than not, we can warn at startup if it's not set, but (IMO) we shouldn't set it ourselves. If we can get the same effect with an iptables rule that only affects kube-proxy's own traffic, then I think that's better than setting a sysctl that will also affect non-kube-proxy traffic. If I understand the situation here correctly, if kube-proxy a drop rule for the invalid conntrack packets, but then the administrator set , then the result would be that conntrack would not mark some packets as invalid, and so our drop rule would just not get hit, and so our drop rule wouldn't interfere with the "better" sysctl-based solution?
Agree in general. We have done it in the past (e.g. route-localnet) and mostly it seems like a not-great idea. Also agree.
If we see this as a known bug, please consider documenting it in - even if latest Kubernetes includes a fix (we can nevertheless point that out!)
I'll fix this in the next few days.
From further googling, it seems like there have been situations in the past where people were setting that sysctl to work around bugs in conntrack, which were later fixed. I think this may be another such situation; it seems like our problem is that there is some state involving dropped or retransmitted packets which linux would cope with fine in the non-conntrack case, but which it doesn't handle in the conntrack case, unless "be_liberal" is set, and that seems like a bug.

Every time I look away from this set of issues/PRs for longer than 15 minutes I forget the exact scenario that causes the bug, but if someone could summarize it very clearly we could try dragging some kernel developers in. (The regression test in the e2e suite creates a packet with an intentionally-bad sequence number, but it's not clear to me what sort of real-world issue that's supposed to be representing.)
Thanks now that was an excellent article! Well written (not TL;DR that is), and IMO perfect technical level. If there is some collection of recommended reading for persons who want to know about K8s networking, this article should be in it.
Hi thank you for the , it's really helpful! I have some questions: , should we update this sentence accordingly ==we use the above workaround, would there any side effect for the cluster network or cluster nodes?
That says "this problem will occur if conntrack is unable to recognize a packet", but it doesn't explain the real-world scenario that causes conntrack to be unable to recognize a packet. If we go to the kernel developers and say "manually injecting random incorrect packets into a conntrack'ed connection causes it to get confused", then they'll likely respond either "don't do that then" or "use ". But if we can tell them something like "if a connection has a duplicated packet after a dropped packet, then conntrack gets confused", then that sounds a lot more like a bug in conntrack that they ought to fix, such that then everything will just work for everyone in the future without needing any DROP rules or sysctls.
So do we need to revert the drop rules? As mentioned in , it now seems that this makes sense.
There are three potential "fixes" here: this is unambiguously a conntrack bug, then we should figure out the details, and get the kernel devs to fix it. However, getting kernel bug fixes into all k8s clusters in the world takes "a long time", so even if it is a kernel bug, we should still think about the other fixes. cluster admins can use . We should advertise this better, but we feel that it would be dubious to have kube-proxy set this flag itself ().

per we could probably tweak the existing rule so that it didn't interfere with non-k8s packets ~and as suggested by the OP of this issue, we could add a similar rule to the INPUT chain (though this would need a bit of further thinking about to make sure we weren't introducing any new conflicts with non-k8s traffic)~.
We could revert the DROP rules and tell people to use the sysctl instead (and actually, probably that's a better answer for this issue than adding a new DROP rule to the INPUT chain is). But we'd need to warn users in advance if we were going to do that, since it would break existing clusters.
Totally agree, This is a better fix for this issue. If this is a kernel bug and it gets fixed someday, we can easily tell the user that you don't need to set anymore. Also, If we revert the DROP rule, which can solve the routing asymmetry issue mentioned in , which is a big fix for us Yes, We need to document it and tell the user. and In order not to break the existing cluster, Do we need a flag to decide if we need the DROP rule? I'll make it default to , and make it consistent with the current logic.
but the change to only apply the DROP rule to pod traffic should also fix that We should not require administrators to choose between "high-traffic TCP services work" and "asymmetric routing works". The current DROP rule is broken. We should either fix it to work with asymmetric routing, or else remove it and tell people who needed it to use the sysctl instead. In the latter case, we might want to make the transition via a feature gate. Then people who need asymmetric routing to work can enable the feature gate in Alpha (to make kube-proxy stop creating the DROP rule), and people who need high-traffic services to work but who didn't get the memo about using the sysctl can disable the feature gate in Beta until they have a chance to update their node configs to set the right sysctl.
Refer to , The DROP rule to pod doesn't fix our case. In our case, the pods of the external cluster subnet failed to access the pods of the cluster subnet to connect due to asymmetric routing, So the packet still matches The DROP rule to pod. Do we need a KEP for this or only a feature gate?

In general, feature gates imply KEPs (and any change that will take multiple releases to complete, where we want to make sure we don't forget about it halfway through, should get a KEP, because we have processes for tracking those). It doesn't have to be a big KEP, and some of the sections of the KEP template will be irrelevant and can be omitted, because you're not introducing a new feature. (See for example.)
We are going to break users, unfortunatly features gates does not have the effect that we are assuming, and we know that all the users have to set the "ipconntracktcpbeliberal" because eventually some connections will be broken. I do no like to set sysctl, but 'm not in favor of replacing the rule and leaving the problem to users that may not even know what a sysctl is ... agree the DROP rule is broken and breaks some esoteric but legit scenarios, but we are already setting sysctls, so I think that if we don't find any alternative to the DROP rule we should try to set the sysctl for the user by default
If we skip a KEP, make sure that the release team know to track the details (comms, docs, etc) as if it were a KEP. This is an important thing to track.
Should we at least triage this as Accepted?
/triage accepted Some alternatives DROP rule and ask users to set sysctl (I'm not in favor per my previous comment and the least surprise principle) DROP rule and we set sysctl set DROP rule if is not set the DROP rule with a flag/config so users can opt-out if don't want to set it (they can decide to use the sysctl or live with that risk) ...
I can accept KEP, but I think this may take multiple releases to complete. how about this? We remove the DROP rule Control the sysctl with a flag or a feature-gate. If it's a flag, I believe it's relatively simple and we can finish it in 1.29. In the case of feature-gate, this may take multiple releases to complete.
If sets the to true by default, we can safely remove the DROP rule
Yes, but if we were doing it that way, that would be because we considered that to be a feature rather than a bug.

We want to avoid breaking working clusters on upgrade. Maybe just having a release note saying "you may need to pass to kube-proxy" is enough. I think we agreed we don't want to change any sysctls except by explicit admin request (because that affects the behavior of non-kubernetes networking as well).
Agree, but we need to think in the users, this is a problem where the fix that solve a sublte and complex bug that may affect 100% of kubernetes users and most probably they don't know they are affected, impact a esoteric network configuration on the cluster networking, and the solution we are offering is "please read the release notes", that we know only 20% of users do ... I think that advanced users can opt-out to the DROP rule if we don't want to mess with the sysctl,
If setting was a no-brainer good idea, then the kernel would just have that behavior by default. There must be some reason why they don't, and so I worry that if we set that flag by default, then we're just going to start breaking some other subset of users. Also, if it turns out that this is all just a conntrack bug, which eventually gets fixed, then we wouldn't want to be setting the sysctl after it got fixed. But at that point it would be likely that there were some users unknowingly relying on the sysctl for some other side effect.
We could potentially have an iptables rule to count invalid conntrack packets, and expose the count as a metric
I completely and absolutely agree. we should not make that sysctl a default in kube-proxy ... what if we read the sysctl and install the DROP rule only if it is not set? Users that apply the sysctl we no longer have the DROP rule
/assign let me send a PR , I think that approach covers all cases
FWIW, my kernel conntrack informant tells me that there really aren't any bad side effects of setting it ("I don't see anything that could break by enabling this mode"), and in fact OVN sets it unconditionally on hosts it's running on. He also tells me that the bug (at least in ) is fixed in kernel ~6.2~ 6.1 ("").

So I'm not sure if that means "we should just go ahead and set the sysctl because it's harmless" or "we should just keep it like it is now and not set the sysctl since it won't be needed in the future"... (NB: if the latter we still need a docs update to go along with this PR, telling admins about the sysctl.)
If we can determine that the kernel fixes the issue in a certain version, then I think we can do this: Do not install the DROP rule.If the kernel version of the host is greater than the safe kernel version, we do not set sysctl.If less than, we have a flag to set sysctl. I'll make it default to 1. (refer to ), And let the user know that we may remove this sysctl in the future.
Exactly my thought. I made this mistake with route_localnet, and we still pay for it. Let me do some internal outreach too, see if I can get a concurring opinion?
Kernel ppl here concur: "I say : if kernel can not be fixed, set tcpbeliberal to one " "having Kubernetes set net.netfilter.nfconntracktcpbeliberal=1 always SGTM, since you want it to work on any kernel version" argues for a flag, default to true (or a hegative flag, defsult false) which seems a bit paranoid, but probably smart :) Are we all in agreement? Who wants to do the PR?
already has a PR to add to the existing set of configurable conntrack sysctls in kube-proxy () so it's just a matter of flipping the default
If the sysctl defaults to true, I think this still keeps backward compatibility, So we don't need the DROP rule anymore.
I just realized that we need this to be a flag because there are cases that run kube-proxy on environments with readonly /sys subsystem, so trying to set it always will fail will and make kube-proxy unusable for those environments
if people does not set it, they will hit the bug until they use kernel 6.1
I mean this sysctl defaults to true in kube-proxy, I can't imagine why users would need to set it to false unless the kernel version is safe (6.1). Of course, we can also document that the user that it is better not to do this.

There are cases where we CAN'T set sysctls but should not hard-fail kube-proxy. On Wed, Sep 6, 2023, 11:25 PM Cyclinder
after setting the jobs for nftables and hitting the conntrack bug continuously I think we should default the flag to true and set the sysctl nfconntracktcpbeliberal to 1 , We are already setting some sysctl and for rootless or other environment they can always opt out, this is what we do in kind
Sounds OK to me. Risk should be ~zero, right?
Yes, I assumed the risk is when to 1, we do not set it by default, so default to 0(see ). I'd be happy with it, Can I have this one?
Talked with offline, I don't have very strong arguments to make this enable by default , so let's wait for more user feedback, nftables will move to beta this release, we can revisit if we have more feedback from users
we are considering turning on by default in our kube-proxy 1.29+ deployments. Based on the recent discussion here that seems fine and the risk should "be ~zero"(), correct? And you are all even discussing turning this on by default in kube-proxy in some future version? One thing I'm a little confused on: is doing the same thing as turning on ? So by setting you're just turning on behavior that will be on by default in 6.1. Did I get that right?
Not exactly. affects a handful of situations. 6.1 only "normalized" one of them. The general intent of is something like "make conntrack work better with packets from broken TCP implementations". But the netfilter devs realized that one of the places it was being used was actually something that could happen due to totally normal TCP packet drops/retransmissions. And so in 6.1+, that particular situation is just handled correctly automatically, whether is enabled or not. My argument against this is that once we do it, we have to keep doing it forever, because people might start unwittingly depending on the other side effects of that sysctl. And particularly given that people never hit the bug anyway.

(It seems like we got more reports of the "my cluster is broken because of the DROP rule" bug than we ever got of the "my cluster is broken because of conntrack invalid packet handling" bug.) day once everyone has moved to 6.1 the bug will be gone forever ISTM that it's better to still not set it by default.
Thanks for the clarification on the 6.1 bugfix That's true and they probably will. I'm just trying to leverage some of the earlier discussion here to make a decision about the safety of flipping this to true, as it seems like the concern was not too high / basically nil.


Question: envArray: []string{ "WITH_OUT_VALUES=", }, <del> expected:  []api.EnvVar{}, expectErr: true, </del> <ins> expected: []api.EnvVar{ { Name:  "WITH_OUT_VALUES", Value: "", }, }, expectErr: false, </ins> test:      "test case 3", }, {

It should be possible to pass empty environment variables to pods run by env (to clear an env var set inside the image, for example). is valid syntax for "set env var FOO to empty string".


Question: } errs = append(errs, err) } <del> if len(infos) == 0 { </del> <ins> if len(infos) == 0 && len(errs) == 0 { </ins> outputEmptyListWarning(errOut) }

Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master What happened: is printed when error. What you expected to happen: Only error message was printed.
I believed this has been fixed
, it seems not; in , it's not necessary to print if errs is not empty. Refer to the PR on the proposal :).
ok, seems other issue.


Question: }) ginkgo.By("Giving the Kubelet time to start up and produce metrics") <del> gomega.Eventually(ctx, getKubeletMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) </del> <ins> gomega.Eventually(ctx, getKubeletMetrics, 2*time.Minute, 10*time.Second).Should(matchResourceMetrics) </ins> ginkgo.By("Ensuring the metrics match the expectations a few more times") <del> gomega.Consistently(ctx, getKubeletMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) </del> <ins> gomega.Consistently(ctx, getKubeletMetrics, 2*time.Minute, 10*time.Second).Should(matchResourceMetrics) </ins> }) ginkgo.It("should not report any admission failures when the topology manager alignment is expected to succeed", func(ctx context.Context) {

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node started to fail since 09-13.
/cc
/assign
/cc
/triage accepted /priority important-soon it's indeed failing too often
/reopen The test is still failing: Now timing out at 120s i.e. 2 minutes as we increased the default timeout . We need to understand the root cause and get to the bottom of this.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Snippet from the build log () corresponding from this failure We see from the above logs that: node has two numa nodes, 8 cpus per NUMA so the root cause of the failure is not because the tests are running on a node that doesn't have multi-numa support. is started and the health check fails a couple of times but eventually succeeds. pod is created with 9 cpus (cpusperNUMA +1 i.e. 8+1) which is expected to fail with Topology Affinity Error as we cannot allocate resources for such a pod from a single NUMAnode. We fail to get matching metrics. It is not clear if the check fails because we don't get kubelet metrics at all or the expected metrics but based on point 6 below it seems like it is because of latter. We can add additional logs for more clarity here. each is executed and Kubelet is restarted and health check is done. are collected from the namespace corresponding to the test pod which shows that the pod started even though it was expected to fail with Topology Affinity Error. We need to understand why the pod got successfully created even though it requested CPUs that couldn't be allocated from a single NUMAnode.


Question: package validation import ( <ins> "fmt" </ins> "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/meta" "k8s.io/kubernetes/pkg/api/unversioned" apiutil "k8s.io/kubernetes/pkg/api/util" "k8s.io/kubernetes/pkg/apimachinery/registered" </ins> "k8s.io/kubernetes/pkg/util/validation" "k8s.io/kubernetes/pkg/util/validation/field" )

ValidateEvent() in checks for "Node" specifically, but should really check for "any non-namespaced object.
I'd like to fix this.
Harder than it sounds, because events may reference 3rd party objects for which we don't know this piece of information...


Question: if err != nil && err != ErrNotFound { return nil, fmt.Errorf("Error getting floating ip for port %s: %v", portID, err) } <del> if floatIP == nil && floatingPool != "" { </del> <ins> if floatIP == nil && floatingPool != "" && !internalAnnotation { </ins> glog.V(4).Infof("Creating floating ip for loadbalancer %s port %s", loadbalancer.ID, portID) floatIPOpts := floatingips.CreateOpts{ FloatingNetworkID: floatingPool, PortID:            portID, } <ins> loadBalancerIP := apiService.Spec.LoadBalancerIP if loadBalancerIP != "" { floatIPOpts.FloatingIP = loadBalancerIP } </ins> floatIP, err = floatingips.Create(lbaas.network, floatIPOpts).Extract() if err != nil { return nil, fmt.Errorf("Error creating LB floatingip %+v: %v", floatIPOpts, err)

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently openstack cloud provider just support for specifying internal LoadBalancerIP. I think that is unreasonable, actually user just use a floating ip to access a LoadBalancer service. So it is necessary to spport for specifying external LoadBalancerIP. What you expected to happen: Support for specifying external LoadBalancerIP on openstack Anything else we need to know?: Currently LoadBalancerIP can be specified in service.Spec.LoadBalancerIP, we can add annotation to distinguish between internal and external.
/sig openstack /assign I'd like to work for it.


Question: if expiresOn == "" { return nil, fmt.Errorf("no expiresOn in cfg: %s", cfgExpiresOn) } <ins> tokenAudience := resourceID </ins> if ts.configMode == configModeDefault { <del> resourceID = fmt.Sprintf("spn:%s", resourceID) </del> <ins> tokenAudience = fmt.Sprintf("spn:%s", resourceID) </ins> } return &azureToken{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When using kubectl, authentication fails with the below error For some reason kubectl is adding a lot of s to the start of the SPN - this can be observed in What you expected to happen: Authentication works as in previous versions of Kubectl How to reproduce it (as minimally and precisely as possible): Create a cluster using Azure AD authentication. Authenticate from a client and wait until the token expires. Use Kubectl and have it automatically refresh the token; it will fail due to the extra 'spn:' entries. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: on-premise OS (e.g: ): macOS 10.14.6 Kernel (e.g. ): Darwin Hostname 18.7.0 Darwin Kernel Version 18.7.0: Mon Feb 10 21:08:45 PST 2020; root:xnu-4903.278.28~1/RELEASEX8664 x86_64 Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others:
/sig cli
Kubectl version 1.18.0 Kubernetes version 1.16.6
can you confirm whether your original kubeconfig already has in apiserver-id or not? in the mean time, you may try it can convert your kubeconfig from azure module to exec module and do the token thing outside kubectl
At this point if I remove the tokens and try to reauth i get the error described above. I've tried it both ways with apiserver-id having an initially and not, both lead to the same behavior in the end.
i know what's going on now. will prepare a fix. the root cause is that token's target resource (adal.Token.Resource) needs to have prefix and this prefix gets unexpectedly to as well, which gets written back to kubeconfig. So every time kubectl is run, it adds to apiserver-id
/priority important-soon
I'm seeing this with kubectl 1.17.3 and kubernetes 1.16.7.

It's unclear to me what versions are impacted and in what version it is fixed, since only version 1.18 is mentioned.
This still happens on kubectl 1.16.12 and 1.17.8
This still happens on kubectl v1.18.5
same
can you guys share your kubectl version --client and kubeconfig with me? you can find me on kubernetes slack. the slack user name is same as my github handle.
Version 1.17.6 works fine.
Since I was mistaken thinking was still causing this for me, I figured I would just drop a note for others in case it's helpful. Turns out my problem was actually due to and looks to have been resolved as of the v3.3.0 release.
This issue has now reappeared for me - although not necessarily the same root cause. AKS 1.19.0 Authentication works, but it spams frequently.


Question: getReactor:                 nil, onRepeatedHeartbeatFailure: nil, expectErr:                  true, <ins> client:                     fake.NewSimpleClientset(node), </ins> }, { desc: "optimistic lock errors",

Some external controller deleted node object while kubelet was alive Due to ignoring errors in kubelet's lease logic, it created lease without ownerReference set Kubelet doesn't create lease without ownerReference set in any case, even pathological one Some external controller deletes node object Garbage collector properly deletes associated lease object Kubelet tries to update object and it fails on Conflict (saying that UID doesn't match): We go to branch, call backoffEnsureLease and assign result to base, but backoffEnsureLease may return potentially invalid object if ensureLease returned error (which happens when newLeasePostProcessFunc returns error, e.g. when node doesn't exist). We The next loop calls c.newLease on the incorrect base and ignores error in line 181 (so the second attempt to call newLeasePostProcessFunc which also fails with an error) We do UPDATE with incomplete (without effect of newLeasePostProcessFunc) object Given that leases allow CreateOnUpdate a new object, without ownerReference is created: No response <details21.10 </details<details</details<details</details<details</details<details</details<details</details
/sig node
/assign
/cc
/triage accepted need to understand what is the right behavior here
We are in trouble for this problem too. I think It’s exactly as said. Lease has the special update endpoint which can create the new lease objects not only update them. If it receives a request with a new lease object which has non-nil UID field, returns a conflict error. This situation can be occured in the function when a lease object is deleted. The lease object is deleted when the node resource is deleted in the result of GC. When received a conflict error in updating a lease object and the parent node object doesn't exist, , and continue next loop. In the next loop, the function returns the new lease object without ownerReference and UID. As a result, it is created unexpectively by the special update endpoint.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: glog.Info("The --provider flag is not set.  Treating as a conformance test.  Some tests may not be run.") case "gce", "gke": <del> framework.Logf("Fetching cloud provider for %qrn", framework.TestContext.Provider) </del> <ins> framework.Logf("Fetching cloud provider for %qr", framework.TestContext.Provider) </ins> zone := framework.TestContext.CloudConfig.Zone region := framework.TestContext.CloudConfig.Region

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: is always appending a new line to its message: In some places of code, the message is already has a newline character and it leads to doubled new lines in the logs. What you expected to happen: I'm not sure what is the correct behavior here. One of the options could be to append a new line conditionally.
/sig testing
I'm new to Kubernetes and would like to start contributing. Is it ok if I assign myself this issue?
/assign MorrisLaw
GitHub didn't allow me to assign the following users: MorrisLaw. Note that only can be assigned. <detailsIn response to : /assign MorrisLaw Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can't assign this to myself, as I am not a member yet. But I am still going to take this issue.
I'll help him on reviews so assigning.


Question: set.Insert(key) } } <del> return nil </del> } // deleteFromIndices removes the object from each of the managed indexes // it is intended to be called from a function that already has a lock on the cache <del> func (c *threadSafeMap) deleteFromIndices(obj interface{}, key string) error { </del> <ins> func (c *threadSafeMap) deleteFromIndices(obj interface{}, key string) { </ins> for name, indexFunc := range c.indexers { indexValues, err := indexFunc(obj) if err != nil { <del> return err </del> <ins> panic(fmt.Errorf("unable to calculate an index entry for key %q on index %q: %v", key, name, err)) </ins> } index := c.indices[name]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG Kubernetes version (use ): k8s1.3.5. still exist in 1.6 What happened: when exec threadSafeMap.Index func, the returned items contain nil object. That is to say, index is in threadSafeMap but the corresponding has already been deleted. What you expected to happen: when object is deleted from threadSafeMap, all its index must be deleted. How to reproduce it (as minimally and precisely as possible): Suppose we init threadSafeMap with two indexers: indexFunc1 and indexFunc2. Obviously, indexFunc may return error if the object has something wrong or some other reasons. when we add an object, will first add the object to , then compute the index and add the index into threadSafe.indices. when we delete the object, will first delete the index from indices, then delete it from Suppose IndexFunc1(object) return error and indexFunc2(object) works well. I think this circumstance is absolutely possible to happen. Also we know that the order to exec indexFunc is not decided. Feature of iterating map. When we add object, suppose first exec indexFunc2, and then exec indexFunc1. So the index of indexFunc2 is to indices. But the index of indexFunc1 is not , because indexFunc1(obj) return err. Then we delete the object, suppose first exec indexFunc1. It will return error and IndexFunc2 has no chance to run. So the corresponding index indexFunc2(obj) can't be removed. But then the object is removed from items. This problem may result in subtle problems. For example, when we exec threadSafeMap.Index( ) func the returned list may include nil object. Anything else we need to know The reason is that index is still is threadSafeMap but the corresponding object has already been removed. We must clear all corresponding indexs when delete object. And the reason why index is not deleted is that when for-range map, the order is not decided.
Who can help me? Do i miss something important or misunderstand it ?
, can you share a unit test to show it? I'd like to help on it.
Thank you for your help. I think i have decribed the condition resulting in the bug indetail.

Be simple, because the order of exec indexFunc is random when update indices, and some function don't care the error returned, the obj is deleted but corresponding indices is still in it. NOTE: the obj is deleted but corresponding indices is still in it So when we run index func, the returned list may contain nil object. Because when get obj from threadsafemap with indice, not care whether the obj exist in this map. Then nil object .
. I am sure this problem exists. And i think we can find a better way to deal with it.
/sig api-machinery threadSafeMap is in client-go
/kind bug
Opened a PR with a fix.
In my understanding, should be deterministic, could you elaborate more how your IndexFunc implemented? If an object can not be called on this , then the question is, how could it exist in at the first beginning?
threadSafeStore should tolerate some bad conditions and deal with it gracefully, instead generating strange and confusing phenomenon. ThreadSafeStore can contain mutiple index funcs. If one of them is not implemented very well, or one of objects we pass to it has something wrong, STRANGE AND CONFUSING phenomenon happens, see what I said above.
That's the point I would like to know more, the call on an object should succeed or fail unconditionally(deterministic).
wechat
I also think index function must be a pure function. See my comment on the pr


Question: for key, informer := range f.targetInformers { glog.V(4).Infof("... Closing informer channel for %q.", key) close(informer.stopChan) <ins> // Remove each informer after it has been stopped to prevent // subsequent cluster deletion from attempting to double close // an informer's stop channel. delete(f.targetInformers, key) </ins> } }

The Stop() method in the Federated Informer races with deleteCluster() in how they both try to close (stopChan). Thanks for the heads up
Fixed by Thanks Maru.


Question: - Score: 2 Utilization: 1 `), <del> wantErr: `decoding .profiles[0].pluginConfig[0]: decoding args for plugin NodeResourcesFit: strict decoding error: unknown field "scoringStrategy.requestedToCapacityRatio.shape[0].Score", unknown field "scoringStrategy.requestedToCapacityRatio.shape[0].Utilization"`, </del> <ins> wantErr: `strict decoding error: decoding .profiles[0].pluginConfig[0]: strict decoding error: decoding args for plugin NodeResourcesFit: strict decoding error: unknown field "scoringStrategy.requestedToCapacityRatio.shape[0].Score", unknown field "scoringStrategy.requestedToCapacityRatio.shape[0].Utilization"`, </ins> }, { name: "v1beta2 NodeResourcesFitArgs resources encoding is strict",

With the addition of server side validation in 1.23, it is now possible for a call to successfully decode it’s data and return a non-nil error when there are strict decoding errors (e.g. duplicate/unknown fields). This means that callers of can distinguish between fatal errors (that cause decoding to fail) and strict decoding errors (that still allow the decode to succeed) by inspecting the target object (to see if it was successfully decoded into) and the error itself (to see if it is a ) With the NestedObjectDecoder however, it only returns an error from giving no indication whether a strict decoding error it has returned comes from an internal decode step that short circuited (by not accounting for strict decoding errors) or actually did successfully decode, but also encountered strict decoding errors. We propose taking a few actions to minimize the possibility of erroneous short-circuiting from Document how all nested decoders should only return strict decoding errors if all decoding is successful or return only non-strict decoding (ie. fatal errors). Document this for all known in-tree and out-of-tree . Update in-tree nested decoders to do the above mentioned behavior (which is just the kube-scheduler) Long-term solutions to better guarantee that no erroneous short-circuiting is taking place could include deprecating the current interface in favor of one that returns both the target object that was decoded into along with an error return a special error type specific to indicating that all internal decoding was successful.
/sig api-machinery
/triage accepted


Question: job.Spec.Template.Spec.Containers[0].Command = []string{"/bin/sh", "-c", "exit $(( $RANDOM / 16384 ))"} case "failOnce": // Fail the first the container of the pod is run, and <del> // succeed the second time. Checks for file on emptydir. </del> <ins> // succeed the second time. Checks for file on a data volume. </ins> // If present, succeed.  If not, create but fail. <del> // Note that this cannot be used with RestartNever because // it always fails the first time for a pod. </del> <ins> // If RestartPolicy is Never, the nodeName should be set to // ensure all job pods run on a single node and the volume // will be mounted from a hostPath instead. if len(nodeName) > 0 { randomDir := "/tmp/job-e2e/" + rand.String(10) hostPathType := v1.HostPathDirectoryOrCreate job.Spec.Template.Spec.Volumes[0].VolumeSource = v1.VolumeSource{HostPath: &v1.HostPathVolumeSource{Path: randomDir, Type: &hostPathType}} // Tests involving r/w operations on hostPath volume needs to run in // privileged mode for SELinux enabled distro, while Windows platform // neither supports nor needs privileged mode. privileged := !framework.NodeOSDistroIs("windows") job.Spec.Template.Spec.Containers[0].SecurityContext.Privileged = &privileged } </ins> job.Spec.Template.Spec.Containers[0].Command = []string{"/bin/sh", "-c", "if [[ -r /data/foo ]] ; then exit 0 ; else touch /data/foo ; exit 1 ; fi"} } return job

Which jobs are flaking: pull-kubernetes-e2e-kind Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig apps
Failure at
Hi I looked into the failures and think it was because the test required the job succeeds 3 times with pseudorandom 50% failure chance within 15 minutes, while there is an exponential back-off delay (10s, 20s, 40s …) capped at 6 minutes before recreating failed pods. As 7 consecutive failures (1/128 chance) could exceed the timeout, the test failed intermittently because of "timed out waiting for the condition". For example, in the failure of , the job pods succeeded and failed in the following orders: After the last failure at , the delay has increased to 640s so no pod was created any more before the test failed at . I think it can be fixed by using a deterministic approach to make the job pods succeed and fail alternately, and its executing time can be reduced to tens of seconds stably. Assigning myself. /assign


Question: if !reflect.DeepEqual(successCase.expectedParts, apiRequestInfo.Parts) { t.Errorf("Unexpected parts for url: %s, expected: %v, actual: %v", successCase.url, successCase.expectedParts, apiRequestInfo.Parts) } <ins> if e, a := strings.Split(successCase.url, "?")[0], apiRequestInfo.URLPath(); e != a { t.Errorf("Expected %v, got %v", e, a) } </ins> } errorCases := map[string]string{

The apiserver proxy rewrites links to the form "/proxy/ns/default/...": But this form stopped working recently, seemingly around The "/proxy/ns/default/..." form always returns "404: Page Not Found", similar to the problem seen in If you remove "/ns/default" from the URL, it works.
/cc
I can investigate this as well.
I see the issue, sending a PR.
The proxy path needs to be built with knowledge of the apiversion of the incoming proxy request, and it needs to be updated to use the knowledge from the restmapper. This is a little trickier than I have time for right now, but it should definitely not be hardcode appending the /ns form. Not clear why this is not also caught by our test case. I can try and look at this more in the AM unless someone else wants to take a look in interim. /cc
Let me steal this one from you, I think I can fix it.
Have at it. /cc me on your PR
Sorry, I see you've already got a PR out!
No problem. I would be happy if you finished my PR relative to the commit I started. It will be a few days before I can go back to looking at this issue. I think v1beta1/v1beta2 proxy urls in go-restful need to always but the namespace scope in the path as the ultimate fix. I do want the proxy to have access to the restmapper per my PR. ----- Original Message ----- From: "Daniel Smith" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Friday, February 13, 2015 4:56:19 PM Subject: Re: [kubernetes] apiserver proxy stopped accepting namespaces () Sorry, I see you've already got a PR out! Reply to this email directly or view it on GitHub:
Any updates ?
This is breaking monitoring since it requires the proxy. Bumping up the priority to P0.
Sorry for dropping this. Working on a fix.


Question: fi } <del> # TODO(#54017): Remove below logics for handling deprecated network mode field. # `x_gcloud_mode` was replaced by `x_gcloud_subnet_mode` in gcloud 175.0.0 and # the content changed as well. Keeping such logic to make the transition eaiser. </del> function check-network-mode() { local mode="$(gcloud compute networks list --filter="name=('${NETWORK}')" --project ${NETWORK_PROJECT} --format='value(x_gcloud_subnet_mode)' || true)" <del> if [[ -z "${mode}" ]]; then mode="$(gcloud compute networks list --filter="name=('${NETWORK}')" --project ${NETWORK_PROJECT} --format='value(x_gcloud_mode)' || true)" fi </del> # The deprecated field uses lower case. Convert to upper case for consistency. echo "$(echo $mode | tr [a-z] [A-Z])" }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network What happened: is adding logics to handle both the new () and deprecated () fields for making the transition easier. We should eventually get rid of such logics when CIs and most of the developers upgraded gcloud version to = 175.0.0. /assign
/area platform/gce
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten


Question: if pp.timeoutPermit { return framework.NewStatus(framework.Wait, ""), 3 * time.Second } <ins> if pp.allowPermit && pod.Name != "waiting-pod" { return nil, 0 } </ins> if pp.waitAndRejectPermit || pp.waitAndAllowPermit { if pod.Name == "waiting-pod" { return framework.NewStatus(framework.Wait, ""), 30 * time.Second

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When preempting a pod that is a that permit plugins are waiting for , we need send a message(actually send a ) to the , otherwise permit plugins might wait until time out, and do not know the pod has been preempted. Why is this needed: /sig scheduling /kind cleanup /assign
cc
Interesting interaction between the permit and preempt, I will take a look at the PR you sent, thanks!


Question: // important when we start apiserver and controller manager at the same time. var versionStrings []string err = wait.PollImmediate(time.Second, 10*time.Second, func() (bool, error) { <del> versionStrings, err = client.ServerAPIVersions(kubeconfig) return err == nil, err </del> <ins> if versionStrings, err = client.ServerAPIVersions(kubeconfig); err == nil { return true, nil } glog.Errorf("Failed to get api versions from server: %v", err) return false, nil </ins> }) if err != nil { glog.Fatalf("Failed to get api versions from server: %v", err)

IIUC, is expected to wait 10s before it fails when is not up (Ref ). Yet it doesn't. k8s version
/cc
Should the warning about this issue be removed from ?
Nope, it is fixed in v1.1.2(also from HEAD), still suffers in v1.1.1 to my best knowledge
v1.1.2 is the latest, right? If the problem is fixed in HEAD, then the warning about it can be removed from the doc in HEAD, right?
Oh, I see your point. The warning is specifically about version 1.1.1, and the problem remains in that version. If that is accurate then the warning should remain (although an remark that the problem is fixed in 1.1.2 would not hurt).
I had a commit for that in , yet was missing by mistake:-/ I'll add it back then.
Agreed that docs on HEAD should reflect the state on HEAD. If someone wants to look at 1.1 docs, they should do so at
oh I see now that the warning says that the issue is only in 1.1.1 and is fixed in 1.1.2. That LGTM. Thx! I guess we can close this issue now?
this issue was already closed. I generated some confusion under another issue, number , because I read the web instead of HEAD. I have closed that issue too.


Question: return ListProcMounts(procMountsPath) } <del> // IsLikelyNotMountPoint determines if a directory is not a mountpoint. // It is fast but not necessarily ALWAYS correct. If the path is in fact // a bind mount from one part of a mount to another it will not be detected. // It also can not distinguish between mountpoints and symbolic links. // mkdir /tmp/a /tmp/b; mount --bind /tmp/a /tmp/b; IsLikelyNotMountPoint("/tmp/b") // will return true. When in fact /tmp/b is a mount point. If this situation // is of interest to you, don't use this function... func (mounter *Mounter) IsLikelyNotMountPoint(file string) (bool, error) { </del> <ins> func statx(file string) (unix.Statx_t, error) { var stat unix.Statx_t if err := unix.Statx(0, file, unix.AT_STATX_DONT_SYNC, 0, &stat); err != nil { if err == unix.ENOSYS { return stat, errStatxNotSupport } return stat, err } return stat, nil } func (mounter *Mounter) isLikelyNotMountPointStat(file string) (bool, error) { </ins> stat, err := os.Stat(file) if err != nil { return true, err

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We saw this several times: after restarting kubelet, kubelet didn't creating new Pods. The healthz endpoint showed that the was stuck: From the stack trace, the goroutine was stuck in the call of when doing , so no other updates can be processed: Listing the volumes of orphaned Pods on this Node, there was an unresponsive NFS volume, stat it will hang forever, so the above goroutine should be stuck in cleaning up this Pod: The NFS server IP is a service IP, whose backend Pod happened to be scheduled to this Node. However, as was stuck, it didn't handle Pod Add/Update from apiserver and didn't create the backend Pod, hence a dead lock. Searched PR and issue history, it seems there were several ones fixing NFS volume issues but looks like not covering this case: (however reverted) (still open) Is there anyone still working on it? The main syncloop of kubelet shouldn't hang due to an unresponsive NFS volume, leading to no Pods can be created. Kubernetes version (use ): v1.20.5
/sig storage /sig node similar with and And the problem code is in According to and 's comments in , it will be removed later, but I checked the new code in mount-utils. The func is not changed: Possible fix to this would be something like below:
/assign
Hey Quick question: the links that you attached that could be possible fixes, were really helpful! Since the approach to check for wouldn't help here, a timeout based approach seems best, I'm a little unsure about what this timeout period could be for this case, any suggestions on the same?
I have no idea. you may ask sig-node guys in the slack or weekly meetings. Or send a PR and discuss it in the PR. The only nfs related timeout that I can found is 1 minute in
hit again, it‘s easy to reproduce if there is nfs service in the cluster.
/assign /triage accepted
/area node-lifecycle
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle rotten /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue New contributor can follow some advice in .
Hey ! I'm wondering if this issue is still being worked on? I'd like to help if there's room to do so!
I'm not working on it. Thanks for the help.
awesome!
/assign
Hi are you still working on this?

I'm interested in this and please let me know if I can give it a try, Thanks! : )
I'm actually still on it but if you're open to it we can team up? I don't mind sharing
Thanks for reply, I am vary glad to hear that, I think you can continue your work and I would take a look at other issues, if there is anything I can help, please feel free to ping me : )
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Is someone still working on this or I am free to take a stab at this? I have ran into this a few times and trying to fix this cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle rotten [1]:
/remove-lifecycle rotten /remove-label good-first-issue This isn't a great first issue to work on. It requires having a NFS server and instrumenting it to be down to test the fix.
The label(s) /remove-label good-first-issue api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, official-cve-feedlabels -additionallabelslabels -? <detailsIn response to : /remove-lifecycle rotten /remove-label good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-good-first-issue
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the PR is closed You can: Mark this PR as fresh with Close this PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Does this problem still exist?
/triage needs-information
/remove-lifecycle stale
/triage accepted cancel
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accepted cancel Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
remove-triage accepted
/remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
statx with ATSTATXDONT_SYNC flag will not stuck in syncLoop. how about replace stat with statx?


Question: func() ([]byte, error) { return []byte(iptablesRestoreCmd + version), nil }, }, RunScript: []fakeexec.FakeRunAction{ <del> func() ([]byte, []byte, error) { return []byte(output), nil, nil }, func() ([]byte, []byte, error) { return nil, nil, &fakeexec.FakeExitError{Status: 1} }, </del> <ins> func() ([]byte, []byte, error) { return []byte(output), []byte(stderrOutput), nil }, func() ([]byte, []byte, error) { return nil, []byte(stderrOutput), &fakeexec.FakeExitError{Status: 1} }, </ins> }, } fexec := fakeexec.FakeExec{

On some systems, iptables-save outputs the following warning to stderr: Note that the error message includes a newline. Because kube-proxy captures both stdout and stderr when calling , the newline might cause rules to be garbled. For example, the output might contain the following line: The parser treats this as a chain line, and panics, as predicted in
/sig network
FYI
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
Just FYI here's another example of garbled output:
Anyway, fix is in Please review.
/assign
/remove-triage unresolved