Question: fakeremote "k8s.io/kubernetes/pkg/kubelet/cri/remote/fake" "k8s.io/kubernetes/pkg/kubelet/eviction" "k8s.io/kubernetes/pkg/kubelet/images" <ins> "k8s.io/kubernetes/pkg/kubelet/kuberuntime" </ins> "k8s.io/kubernetes/pkg/kubelet/lifecycle" "k8s.io/kubernetes/pkg/kubelet/logs" "k8s.io/kubernetes/pkg/kubelet/network/dns"

spans are not being parented to other spans - propagation of context seems broken. More context: spans should be parented to other spans enable kubelet tracing at v1.28 and hook up a tracing backend. No response <detailsv1.28 at least </details<detailsNot sure this matters. </details<details</details<details</details<details</details<details</details
/sig instrumentation /assign
/triage accepted
!
repro steps:
/assign Thanks for helping to look into this! Feel free to post the findings here you shared with me on slack
GitHub didn't allow me to assign the following users: fatsheep9146. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks! I will post this ASAP
The broken issue is due to the pr , which try to propagate context brought by pod worker to syncPod function in order to let the syncPod can be interrupted in time by other asynchronous routines, for example eviction. But in the line we did not pass the context which contain the parent span and trace into function, so the trace is broken. So I want to fix this by passing the into , which is also desired in comment But passing the ctx into syncPod may have something other to do as described in comments, so I tried to fix as comments advised.
cc


Question: switch { case err == ErrExit: handleErr("", DefaultErrorExitCode) <del> case apierrors.IsInvalid(err): details := err.(*apierrors.StatusError).Status().Details </del> <ins> case isInvalidReasonStatusError(err): status := err.(*apierrors.StatusError).Status() details := status.Details </ins> s := "The request is invalid" if details == nil { <ins> // if we have no other details, include the message from the server if present if len(status.Message) > 0 { s += ": " + status.Message } </ins> handleErr(s, DefaultErrorExitCode) return } if len(details.Kind) != 0 || len(details.Name) != 0 { s = fmt.Sprintf("The %s %q is invalid", details.Kind, details.Name) <ins> } else if len(status.Message) > 0 && len(details.Causes) == 0 { // only append the message if we have no kind/name details and no causes, // since default invalid error constructors duplicate that information in the message s += ": " + status.Message </ins> } if len(details.Causes) > 0 { errs := statusCausesToAggrError(details.Causes) handleErr(MultilineError(s+": ", errs), DefaultErrorExitCode)

kubectl apply displays different errors in v1.21 (and 1.22) vs v1.23 (and above) for the same webhook response, with the error message in v1.23 not being helpful at all. For example, if a webhook returns a status with code is not equal to / the and fields are not populated In v1.21.0, we see the following error. This is a helpful error message with plenty of context. In v1.23.5, for the same webhook status, we'll see: This is not helpful at all. To get an equivalent helpful error message in v1.23.5 Make a webhook return a status with code with or not populated, or not equivalent to . Looking at the code, it looks like this was a consequence of Since this PR was introduced in v1.23, this issue would exist for v1.22 as well. kubectl first checks whether the error message is (in both and ) The definition of was changed in . In 1.21 (and 1.22) just reason "Invalid". Given that the webhook status does not include a , the is executed and kubectl shows a helpful error message. In 1.23 (and above) if OR if the status code is , which is true for the webhook status. Since it’s true, this is executed. is nil, so we just see the response. This error message is not helpful. We should probably update how kubectl handles "invalid" errors and ensure that they display the field too (instead of just restricting to check the ). Verified with v1.21.0 and v1.23.5. Have not verified this for v1.22, but it should also affect v1.22 due to the above reasons. N/A No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign
Hi So this purely a client-side issue, right?

The server is still sending back the exact same response/data structure? Thanks!
FYI, I posted in Kube slack related to this. We were hoping to alter our webhook behavior based on the originating user agent by passing the into the Go context given to our webhook server's request handler. However, it appears we are not able to access the originating user-agent info. As indicated in the Slack thread: In other words, the sooner this issue is fixed, the better.
checked with nikhita offline, will reroute the issue /assign /unassign
fix open in


Question: // When GetVolumesToReportAttached is called by node status updater, the current status // of this node will be updated, so set the flag statusUpdateNeeded to false indicating // the current status is already updated. <del> asw.updateNodeStatusUpdateNeeded(nodeName, false) </del> <ins> if err := asw.updateNodeStatusUpdateNeeded(nodeName, false); err != nil { glog.Errorf("Failed to update statusUpdateNeeded field when getting volumes: %v", err) } </ins> } return volumesToReportAttached

was the fix in sufficient, or do you want to bubble the error past SetNodeStatusUpdateNeeded?
do you want this in v1.7?
can you retitle this to be meaningful or close it?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I checked the sources, now returns as requested here. /close


Question: "":              "Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key <topologyKey> matches that of any node on which a pod of the set of pods is running", "labelSelector": "A label query over a set of resources, in this case pods.", "namespaces":    "namespaces specifies which namespaces the labelSelector applies to (matches against); null or empty list means "this pod's namespace"", <del> "topologyKey":   "This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. For PreferredDuringScheduling pod anti-affinity, empty topologyKey is interpreted as "all topologies" ("all topologies" here means all the topologyKeys indicated by scheduler command-line argument --failure-domains); for affinity and for RequiredDuringScheduling pod anti-affinity, empty topologyKey is not allowed.", </del> <ins> "topologyKey":   "This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed.", </ins> } func (PodAffinityTerm) SwaggerDoc() map[string]string {

The field is marked as optional: The comment also describes a use case for empty topologykey: But the validation rule says it's required:
cc
cc could you take a look? Thank you. The validation rule is by
the release note of should go into the "action-required" section of the , but it's currently in "Other notable changes".
/assign
It seems that we should also update the external api type comments. Right?
I will look into it today.
If you didn't work on this, I am pleasure to send PR to fix this. :)
yeah sure. thanks.


Question: } var left, right interface{} <del> if len(lefts) != 1 { </del> <ins> switch { case len(lefts) == 0: continue case len(lefts) > 1: </ins> return input, fmt.Errorf("can only compare one element at a time") } left = lefts[0].Interface()

On , will fail to evaluate a filter when the member does not exist: For instance, if I were to execute Where not all in that namespace had the annotation, this would fail with the error message: This is confusing as it leads me to think more than one item was selected, so while it is not technically incorrect the error message is misleading. I would furthermore expect to more gracefully handle this case -- a missing here means that the filter should fail, not error.
What do you mean by ? How is it different from ?
As in, don't add that item to the list of items on which the following access is made. When I do: And there is no annotation on , I just get nothing back -- no error, just nothing. Similarly, i would have expected the filtered expression to give me the for all of the pods that have an annotation with value . That, anyway is what I understand the JSONPath syntax to mean. When you write You are saying:
cc since you authored and I'm not sure how we want to fix this.
Taking a look now
it looks like you need to do this (note that the annotations portion is wrapped by ): I wrote a unit test and it worked as expected. PTAL!
I'm pretty sure that was a typo on my end but let me check -- do you have a reproducer script?
I see an entirely different error today... weird:
Ok so you have 1 object with annotations, and 1 without. Let me look into that case.
Let me add another annotation to
No difference if both have annotations:
Ok, I'll have to test using kubectl
I'm the Origin bundled but I don't think there are any patches on JSONPath there:
I'm able to reproduce in the unit test when 1 object has annotations and the other does not, if allowMissingKeys is true for the jsonpath parser.
I see. Let me know if I can be of more help.
I think I have a fix. PR coming soon.
Possible


Question: import ( "fmt" <ins> "sync" </ins> "time" v1 "k8s.io/api/core/v1"

pull-kubernetes-unit, see example failure in It appears to be a new test in No response No response No response /sig
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Repro's locally:
/cc can you PTAL? It seems related to new test in
flaking regularly -
actually, I'm not sure this is a test-only race... it looks like self mutates , and that is called in a goroutine in parallel with other methods that access rootDirectory: I suspect this has always been a race, and the unit test is exercising / catching it more
moving the filepath.Clean to init time: and changing the check in setupDataDirs to be a validation rather than a mutation: might resolve the immediate race
/assign
is open to fixing it. The data race we find here should be fixed. However, is another one that we need to fix. I add a new commit to fix that as well.
did you see that race as well or did you just see the issue opened? that was in a code path that should be feature gated so I wouldn't have expected it to trigger in the kubemark unit tests
I may mixed problems together. Yes. will not trigger in the kubemark unit tests. Using stressing, I find the data race cannot fix the problem completely. Then I saw and cherry-picked the fix to my PR. But the hollowNode test still flakes and I looked into it and find it is the volumeManager problem. is just another potential data race in .

the rootDirectory and volumeManagerSyncTime race fixes are easy to reason about and fix code that looks like it could really race in currently enabled-by-default code looks like a real race in disabled-by-default code. I'll defer to kubelet folks and whether to mix that fix into
I think I should drop that fix. This will make things more clear.


Question: defer f.statusLock.Unlock() return f.removeRuntime[uid] } <ins> func (f *fakePodWorkers) setPodRuntimeBeRemoved(uid types.UID) { f.statusLock.Lock() defer f.statusLock.Unlock() f.removeRuntime = map[types.UID]bool{uid: true} } </ins> func (f *fakePodWorkers) ShouldPodContentBeRemoved(uid types.UID) bool { f.statusLock.Lock() defer f.statusLock.Unlock()

pull-kubernetes-unit TestVolumeUnmountAndDetachControllerDisabled It failed on but after that the test is passing DATA RACE It seems that the data race is caused by below code It failing of test is Flaky, it was not reproducible on my local machine too, I have used below command to run the test output on my local machine I would like to work on this issue, is it possible to reproduce this issue locally?
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
SIG Testing owns test frameworks, CI infrastructure, and other things to support testing. But this test is owned by SIG storage I believe since it tests their code, as we own tests that test our code.


Question: klog.Errorf("Failed to ensure rule to drop packet marked by %v in %v chain %v: %v", KubeMarkDropChain, utiliptables.TableFilter, KubeFirewallChain, err) return } <ins> // drop all non-local packets to localhost if they're not part of an existing // forwarded connection. See #90259 if !kl.iptClient.IsIPv6() { // ipv6 doesn't have this issue if _, err := kl.iptClient.EnsureRule(utiliptables.Append, utiliptables.TableFilter, KubeFirewallChain, "-m", "comment", "--comment", "block incoming localnet connections", "--dst", "127.0.0.0/8", "!", "--src", "127.0.0.0/8", "-m", "conntrack", "!", "--ctstate", "RELATED,ESTABLISHED,DNAT", "-j", "DROP"); err != nil { klog.Errorf("Failed to ensure rule to drop invalid localhost packets in %v chain %v: %v", utiliptables.TableFilter, KubeFirewallChain, err) return } } </ins> if _, err := kl.iptClient.EnsureRule(utiliptables.Prepend, utiliptables.TableFilter, utiliptables.ChainOutput, "-j", string(KubeFirewallChain)); err != nil { klog.Errorf("Failed to ensure that %s chain %s jumps to %s: %v", utiliptables.TableFilter, utiliptables.ChainOutput, KubeFirewallChain, err) return

What happened: is set by in both iptables and ipvs mode. This causes a security issue; A host on the local segment can actually route 127.0.0.1 to a k8s-node and access ports that really should be local. Linux doc; What you expected to happen: Make configurable and set only when needed. How to reproduce it (as minimally and precisely as possible): On a machine on the same network segment as the k8s nodes; The example uses port which is opened by kubelet on 127.0.0.1 only; I really can't the the connect working myself on plain linux with the instruction above. I got this info from others who claim they can. What I do see though is that the syn-ack is sent back, so I guess some spy program can exploit this; If you set manually on the target node the syn-acks will not be sent back; Anything else we need to know?: Related issue Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Kvms on Ubuntu OS (e.g: ): Kernel (e.g. ): Install tools: "from scratch" Network plugin and version (if this is a network-related bug): (any) Others:
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Are there any reasons to set in the ipvs proxier? If not, I will make a PR to remove it. Considering the problems described in I don't think it makes any difference.
To set set the enabler for proxy-mode=iptables;
/cc
Hi all, Just to add a confirmatory post on this. I've just run up an experiment which looks like this single node Kubeadm 1.18.2 cluster with Weave as CNI, running on Ubuntu 18.04.3 server in VMWare Workstation on Linux. Straight install no changes from the standard process. Server IP is 192.168.41.208client is ubuntu 18.04.3 in VMWare workstation on Linux. Client IP is 192.168.41.248 On the client I ran (as per the original post) The listening ports on the server are Listening ports on the client are From the client run Response received : - So looks like it's possible to get app.

level traffic back, over the local LAN. Obviously if there's any services that are assuming some level of security based on listening on localhost only, this may not apply.
I don't have the full context for this but in doing some digging I think this was mostly because iptables had it. I wouldn't be surprised though if disabling it broke some functionality that users depend on. For IPVS, even though we don't permit node port via localhost (yet), it sounds like it still has implications for when the source is from localhost?
This allows some corner cases of in iptables, but honestly I forget the details. I'm baffled why the kernel would respond to localhost traffic when it's clearly martian.
This seems relevant background:
Well - thats what the documentation says it does.. :) "Do not consider loopback addresses as martian source or destination" (from: ) Here are some further explanation and suggestions as to secure use of this (by other people) - that seems useful
I believe is enabled so that pods with HostPort set and services of type NodePort are truly equivalent to host ports. That is to say, I should be able to connect to not know that it's actually handled by a pod. I seem to recall this being lifted directly from libnetwork, but memory is extremely hazy.
Also, none of this works for ipv6; there is no equivalent, as addresses cannot escape their zone.
Yes, that is reported in;
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
The case is connecting to . Let's say that's 8888, pointing to pod 10.0.0.1:9999. packet. PREROUTING rules run, do DNAT, add conntrack entry. packet is decision. normally, 127.0.0.1 would be dropped as a martian, but permits this. The outgoing interface is decided. POSTROUTING rules run, do MASQ, add conntrack entry. packet is now This sets up the conntrack entries for the packet to flow back from the pod to the tcp socket on localhost. You need to get the packet past the martian barrier, though, for this to work.

Can people use 0.0.0.0:NODEPORT ? Activating an insecure sysctls and trying to secure it seems wrong.
BTW does 127.0.0.1:$NODEPORT work with IPVS ? It's not working on my CentOS7 Calico cluster
No it doesn't work with ipvs, please see;
0 doesn't work (it's also local). wondering if we can make it work with ip rule / ip route but without route_localnet
Since accessing localhost:nodeport from the host never worked in IPVS mode (see 's comment), how do you feel about not setting the sysctl (or even setting it to 0)? It will make easy for users to patch their clusters if they run kube-proxy as a daemonset (we plan to do that)
+1 from me as long as we test it and ensure it doesn't break any existing behavior. There were previous feature requests to actually allow localhost:nodeport but given the security implications it sounds like we should avoid it.
I want to re-open this because I want to discuss the "real" fix. Clearly we have disagreement between ipvs and iptables about whether node-ports should work on localhost. I know of at least one use-case for it and I know that, at some point in the past, people were relying on it. But IPVS doesn't support it at all. Does anyone know about out-of-tree implementations? What do they support? I am anxious about shutting this off for iptables, since users MIGHT be relying on it. It's actually kind of "obvious" that it should work. That it DOESN'T is surprising (though apparently not fatally so). Are there any smaller mitigations we can take?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
localhost:nodeport works with Cilium but I am unsure how much load-balancing cilium handles (I deployed the "standard" yaml with ipv6 enabled); Update; localhost:nodeport does not work with Cilium and proxy-mode=ipvs so it seems Cilium only acts as cni-plugin by default.

I am pretty sure does not support localhost:nodeport since it uses and should have the same limitation as in ipvs-mode.
K3s relies on for their forwarding of the endpoint via "ssh", please see But if they want to support ipv6 that does not work so...
Deployed with "all-features" yaml and kube-proxy disabled. Access to localhost:nodeport does not work users who disable do not suffer from the security issue
Cilium's BPF-based kube-proxy replacement supports accessing a NodePort service via , and thanks to BPF it does NOT set (therefore, it's not vulnerable to CVE-2020-8558). See You probably ran on an old kernel version which does not support some features required by the BPF NodePort. You can check to see what the kube-proxy replacement features were enabled.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
I'm going to close it since kubelet a rule to mitigate the problem for kube-proxy iptables and IPVS totally disable it /close reopen if you feel that something is missing
Closing this issue. <detailsIn response to : I'm going to close it since kubelet a rule to mitigate the problem for kube-proxy iptables and IPVS totally disable it /close reopen if you feel that something is missing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // update security rules dirtySg := false var updatedRules []network.SecurityRule <del> if sg.SecurityRules != nil { updatedRules = *sg.SecurityRules </del> <ins> if sg.SecurityGroupPropertiesFormat != nil && sg.SecurityGroupPropertiesFormat.SecurityRules != nil { updatedRules = *sg.SecurityGroupPropertiesFormat.SecurityRules </ins> } for _, r := range updatedRules {

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT /kind bug What happened: Created a service of type on Azure. The service is created successful and a LoadBalancer with public IP is created on Azure. kube-controller-manager goes into a crashloop What you expected to happen: LoadBalancer service receives public IP and is available for traffic How to reproduce it (as minimally and precisely as possible): provision a loadbalancer service on Azure with v1.10.3. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/sig api-machinery /sig azure
Logs of the initial request to create the load balancer service:
/remove-sig api-machinery
fyi
FYI
BTW, this is with an Availability Set.
Update: my was missing: Once I it LoadBalancer services work. Still, we shouldn't panic in the main loop on this.
Thanks I think we can close this issue. Create another for handling errors in both the cloud provider and the controller.
/cc
/assign
Filed to fix this


Question: // startupProbe fails, but is disabled m.prober.exec = fakeExecProber{probe.Failure, nil} msg = "Started, probe failure, result success" <del> expectContinue(t, w, w.doProbe(), msg) </del> <ins> expectStop(t, w, w.doProbe(), msg) </ins> expectResult(t, w, results.Success, msg) }

What happened: liveness/readiness probe fails while pod is terminated. Also it happened only once during the pod termination. The issue started happening after upgrading version to v1.7 from v1.6.X How to reproduce it (as minimally and precisely as possible): execute to delete pod, so status of the nginx-podA1 is changed to , right after that it seems Liveness and Readiness Probe is executed and failed, but only once. Nginx reverse proxy is running in the pod. so I just use method for liveness and readiness Here is my Deployment config. Here is Events log by Environment: Kubernetes version: 1.7.2
/kind bug
/sig node
PING
Since the upgrade to 1.7, it seems our deployment rollouts have a higher failure rate. Occasionally, the pod would come up, but no readiness probe ever gets started. It stays in that state, blocking the entire deployment. I usually have to delete the pod so it is rescheduled, and a new readiness probe is fired to check. I wonder if these are related issues here.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen /remove-lifecycle rotten We see this consistently with all pods who define a liveness or readiness probe. Whenever we roll out a new deployment, the pods who are terminated will emit a failed liveness/readiness probe AFTER they have been terminated.

We have considered adding a hook that just sleeps for 2-3 seconds, but it seems like a band-aid solution to something that should not happen in the first place. Is this an impossible-to-solve race condition between kubernetes moving parts?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Do you still experience this? This happens to us running kubernetes 1.11.4.
On k8s 1.12.7 and this is happening to me as well.
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} still happen. I think just add a phase "Terminateing" in pod, when an pod is in Terminateing just stop check.
I think it still happening in 1.13, why this issue was closed? Logs, !
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
still experience this on kubernetes 1.12.4. delete event and probe failed event so it seems that after the pod is deleted, the probe keeps running.I am wondering if the probe is started before the pod deletion.
/area kubelet
/priority important-soon
Any progress on this issue? It seems like some kind bug.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /assign
GitHub didn't allow me to assign the following users: ashleyschuett. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /remove-lifecycle rotten /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
16.9 here. Same issue.
/assign
I am also facing this issue. We have pods which have a lot of cleanup to do during shutdown, it can take up to 5 mins to terminate gracefully. During this time the livelinessProbe is detecting failure and restarting the pod. not really what we want. I am unable to prevent the service that handles the liveliness check from stopping while the cleanup is happening. It would be better if the pod was immediately removed from the service and the probes stopped while the shutdown is performed. Basically this ends up that k8s never actually is able to terminate the pod.
I will take that point and propose a shutdown probe to API sig.

Same here at 1.18.8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
kubernetes_version 1.19.3. Same issue.
Same here, kubernetes 1.19.4.
/remove-lifecycle rotten
/assign
Should I consider a cherry-pick for 1.20 and 1.19? (maybe 1.18 too?)
we're running 1.16 and being hit by this continuously when some of our elixir apps shutdown so cherry-picking it in 1.18 would at least put it closer in our update path :angel:


Question: // TolerationsTolerateTaintsWithFilter checks if given tolerations tolerates // all the taints that apply to the filter in given taint list. <ins> // DEPRECATED: Please use FindMatchingUntoleratedTaint instead. </ins> func TolerationsTolerateTaintsWithFilter(tolerations []v1.Toleration, taints []v1.Taint, applyFilter taintsFilterFunc) bool { <del> if len(taints) == 0 { return true } </del> <ins> _, isUntolerated := FindMatchingUntoleratedTaint(taints, tolerations, applyFilter) return !isUntolerated } </ins> <del> for i := range taints { if applyFilter != nil && !applyFilter(&taints[i]) { continue </del> <ins> // FindMatchingUntoleratedTaint checks if the given tolerations tolerates // all the filtered taints, and returns the first taint without a toleration func FindMatchingUntoleratedTaint(taints []v1.Taint, tolerations []v1.Toleration, inclusionFilter taintsFilterFunc) (v1.Taint, bool) { filteredTaints := getFilteredTaints(taints, inclusionFilter) for _, taint := range filteredTaints { if !TolerationsTolerateTaint(tolerations, &taint) { return taint, true </ins> } <ins> } return v1.Taint{}, false } </ins> <del> if !TolerationsTolerateTaint(tolerations, &taints[i]) { return false </del> <ins> // getFilteredTaints returns a list of taints satisfying the filter predicate func getFilteredTaints(taints []v1.Taint, inclusionFilter taintsFilterFunc) []v1.Taint { if inclusionFilter == nil { return taints } filteredTaints := []v1.Taint{} for _, taint := range taints { if !inclusionFilter(&taint) { continue </ins> } <ins> filteredTaints = append(filteredTaints, taint) </ins> } <del> return true </del> <ins> return filteredTaints </ins> } // Returns true and list of Tolerations matching all Taints if all are tolerated, or false otherwise.

When trying to see why pods don't start using either kubectl get events or kubectl describe pod , the base message is not detailed enough. Warning FailedScheduling 5s (x4 over 65s) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. What would you like to be : Details of the taint and details of what it didn't tolerate Why is this needed: To get much much better debug information by default when trying to understand why pods can't launch.
/sig cli
/sig scheduling /priority important-longterm
/cc
/remove-sig cli
Those labels are not set on the issue: <detailsIn response to : /remove-sig cli </details
/good-first-issue
/assign starizard
Can I get an idea of what this would involve? I might be able to put some work on it.
I assume is taking this already. The error message is produced in We probably also require changes in to get the name of the missing toleration.
/remove-help


Question: package qos import ( <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/kubernetes/pkg/apis/core"

What happened: A pod that has init container with request for cpu and memory, and containers that make no request for cpu or memory reports a BestEffort QoS class. This is incorrect. What you expected to happen: Containers that make requests independent of lifecycle phase should impact quality of service calculation. Found in 1.12, looks to still be present in 1.14 dev. /cc
/priority important-soon
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
I think this doesn't work very well with the current cpu manager logic. The problem is that a pod can't now have QoS class Guaranteed if it has init containers which don't have matching requests/limits. This means that normal containers in such a pod don't then get dedicated cpus even though the init containers are long gone when the app containers (with matching requests/limits) are finally started. This is quite non-intuitive. I would propose separating at least Guaranteed class calculation so that the init container resource requests wouldn't affect the calculation. What do you think?


Question: return true, nil } <del> // retrieve the previous number of retry previousRetry := jm.queue.NumRequeues(key) </del> // Check the expectations of the job before counting active pods, otherwise a new pod can sneak in // and update the expectations after we've retrieved active pods from the store. If a new pod enters // the store after we've checked the expectation, the job sync is just deferred till the next relist.

What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): $oc create job -f $ oc get job jobname -ojson | jq .status { "active": 1, "failed": 1, "startTime": "2020-07-30T06:45:37Z" } { "active": 1, "failed": 2, "startTime": "2020-07-30T06:45:37Z" } { "active": 1, "failed": 3, "startTime": "2020-07-30T06:45:37Z" } { "active": 1, "failed": 4, "startTime": "2020-07-30T06:45:37Z" } { "conditions": [ { "lastProbeTime": "2020-07-30T06:53:10Z", "lastTransitionTime": "2020-07-30T06:53:10Z", "message": "Job has reached the specified backoff limit", "reason": "BackoffLimitExceeded", "status": "True", "type": "Failed" } ], "failed": 5, "startTime": "2020-07-30T06:45:37Z" } $ oc get job jobname -ojson | jq .spec.backoffLimit 6 Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: 4.5.3 Server Version: 4.4.13 Kubernetes Version: v1.17.1+ Cloud provider or hardware configuration: OS (e.g: ): NAME="Red Hat Enterprise Linux CoreOS" VERSIONID="4.4" OPENSHIFTVERSION="4.4" RHELVERSION="8.2" ID="rhcos" IDLIKE="rhel fedora" ANSICOLOR="0;31" CPENAME="cpe:/o:redhat:enterpriselinux:8::coreos" HOMEURL="" BUGREPORTURL="" REDHATBUGZILLAPRODUCT="OpenShift Container Platform" REDHATBUGZILLAPRODUCTVERSION="4.4" REDHATSUPPORTPRODUCT="OpenShift Container Platform" REDHATSUPPORTPRODUCTVERSION="4.4" OSTREEVERSION='44.82.-0' Kernel (e.g.

): Linux ip-10-0-192-232 4.18.0-193.13.2.el82.x8664 SMP Mon Jul 13 23:17:28 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Others: apiVersion: batch/v1 kind: Job metadata: name: sre-build-test-fail namespace: openshift-build-test spec: backoffLimit: 6 completions: 1 parallelism: 1 template: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: matchExpressions: key: node- operator: In values: infra containers: command: /bin/bash -c | # ensure we fail if something exits non-zero set -o errexit set -o nounset set -o pipefail exit 1 imagePullPolicy: Always name: sre-build-test restartPolicy: Never terminationGracePeriodSeconds: 30 tolerations: effect: NoSchedule key: node- operator: Equal value: infra
/sig apps
not sure but that is the best I can think of at this time
i have noticed this problem also, and sometimes, it seems to work normally.
so I found out that in the test is so this can cause the issue.. it has been there from the beginning of the codes inception.. WDYT?
/assign
Duplicate of . /close
Closing this issue. <detailsIn response to : Duplicate of . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: source "${KUBE_ROOT}/cluster/gce/config-common.sh" CLEANUP_KUBEMARK_IMAGE=${CLEANUP_KUBEMARK_IMAGE:-true} <ins> TEST_CLUSTER_LOG_LEVEL="${TEST_CLUSTER_LOG_LEVEL:-"--v=4"}" # If you want to set up multiple kubemark clusters with different "names", # you should change this env per each start-kubemark.sh invocation. KUBE_GCE_INSTANCE_PREFIX="${KUBE_GCE_INSTANCE_PREFIX:-"default"}" </ins> # NUM_NODES is used by start-kubemark.sh to determine a correct number of replicas. NUM_NODES=${KUBEMARK_NUM_NODES:-10}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Run , will get error: Try to fix this by and rerun get another error: Basically, we should have default value for and . I found that they have default values before, and was deleted accidentally in . Although we can predefine these two variables before running the script, it will be more convenient to have default values, and more friendly to people who are not familiar with this script. What you expected to happen: We should be able to run without setting any environment variable manually. How to reproduce it (as minimally and precisely as possible): /cc /sig testing /sig cluster-lifecycle /assign
I have opened a PR to fix this.


Question: build-integration-test-image: build cp -r $(TEMP_DIR) $(TEMP_DIR)_integration_test cp Dockerfile $(TEMP_DIR)_integration_test/Dockerfile <del> cd $(TEMP_DIR)_integration_test && sed -i.bak 's|BASEIMAGE|golang:$(GOLANG_VERSION)|g' Dockerfile docker build --pull -t etcd-integration-test $(TEMP_DIR)_integration_test </del> <ins> docker build  --pull  -t etcd-integration-test  --build-arg BASEIMAGE=golang:$(GOLANG_VERSION)  --build-arg RUNNERIMAGE=$(RUNNERIMAGE)  $(TEMP_DIR)_integration_test </ins> integration-test: docker run --interactive -v $(shell pwd)/../../../:/go/src/k8s.io/kubernetes$(DOCKER_VOL_OPTS) -e GOARCH=$(ARCH) etcd-integration-test

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Building the etcd integration-test-image fails with the following err: Failure Log: Expected the image build to pass. k/k cluster/images/etcd build-integration-test-image Change that probably broke it: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig etcd
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig etcd Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
/sig api-machinery
/cc /cc
/cc
/assign /triage accepted
is working on this in .


Question: Field: fields.Everything(), } <del> // MatchValue defines a pair (<index name>, <value for that index>). </del> <ins> // MatchValue defines a pair (`<index name>`, `<value for that index>`). </ins> type MatchValue struct { IndexName string Value     string


Question: proxy := httputil.NewSingleHostReverseProxy(&url.URL{Scheme: h.Location.Scheme, Host: h.Location.Host}) proxy.Transport = h.Transport proxy.FlushInterval = h.FlushInterval <del> proxy.ServeHTTP(w, newReq) </del> <ins> proxy.ServeHTTP(maybeWrapFlushHeadersWriter(w), newReq) } // maybeWrapFlushHeadersWriter wraps the given writer to force flushing headers prior to writing the response body. // if the given writer does not support http.Flusher, http.Hijacker, and http.CloseNotifier, the original writer is returned. // TODO(liggitt): drop this once https://github.com/golang/go/issues/31125 is fixed func maybeWrapFlushHeadersWriter(w http.ResponseWriter) http.ResponseWriter { flusher, isFlusher := w.(http.Flusher) hijacker, isHijacker := w.(http.Hijacker) closeNotifier, isCloseNotifier := w.(http.CloseNotifier) // flusher, hijacker, and closeNotifier are all used by the ReverseProxy implementation. // if the given writer can't support all three, return the original writer. if !isFlusher || !isHijacker || !isCloseNotifier { return w } return &flushHeadersWriter{w, flusher, hijacker, closeNotifier} } type flushHeadersWriter struct { http.ResponseWriter http.Flusher http.Hijacker http.CloseNotifier } func (w *flushHeadersWriter) WriteHeader(code int) { w.ResponseWriter.WriteHeader(code) w.Flusher.Flush() </ins> } // tryUpgrade returns true if the request was handled.

What happened: Prior to 1.14, when you created a watch on a resource coming from an aggregated API, you would instantly obtain the object, and could stop it when you want. Since 1.14, when you do the same, the method returns when the server sends its first event. Resources directly hosted by the central API-server (core resources, crds, etc.) are not touched by this issue. When you use higher level APIs like informers, situations like this can make your code hang, as attempts to stop the informer seem ignored (the informer cancellation logic only works once the underlying watch.Interface has been retrieved), and your code hang. This is hitting , and a workaround has been found here: However this change of behavior means that current versions of the CLI can hang when deploying Kubernetes workloads. What you expected to happen: As with resources hosted by the central API server, the watch.Interface object should be instantly returned How to reproduce it (as minimally and precisely as possible): I'll try to setup an automated test case for that with compose-on-kubernetes. In the mean-time: deploy a kube 1.14 cluster with kind / minikube / ... deploy compose-on-kubernetes on it (see ) download the latest official docker cli release and try to deploy a simple compose file to kubernetes using Anything else we need to know?: After digging a little with a debugging http proxy, I saw that the big difference seem to be that: prior to 1.14, when such a watch is issued, the response headers are immediately sent to the client with 1.14, on central-API resources, it does not change with 1.14, on aggregated API resources, the reponse headers are sent with the first event Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Using Kind, also happen with internal Docker Desktop builds OS (e.g: ): Both Kind, and Linuxkit Kernel (e.g. ): Install tools: Others:
/sig api-machinery
cc
a fix similar to might be required, where headers are flushed after write
looks like watch is flushed here:
please note that the problem manifests itself only for aggregated APIs. My guess is that there is something in the proxy created by kube-aggregator that does not propagate the flush down to the client.

Not seeing issues there, it explicitly flushes on an interval as well: Also, that code hasn't changed since 1.13 Will try to reproduce locally
Actually, looks like the golang reverse proxy reworked flushing in go1.12: the flushing now takes effect on each body write, but I don't see anything forcing a flush after headers are written I'll take the issue upstream /assign
reported upstream as
Really good catch!
workaround in until we pick up a fix from golang
picked to 1.14.1 in


Question: type StatFunc func(string) (os.FileInfo, error) type GlobFunc func(string) ([]string, error) <del> func waitForPathToExist(devicePath string, maxRetries int, deviceTransport string) bool { </del> <ins> func waitForPathToExist(devicePath *string, maxRetries int, deviceTransport string) bool { </ins> // This makes unit testing a lot easier return waitForPathToExistInternal(devicePath, maxRetries, deviceTransport, os.Stat, filepath.Glob) } <del> func waitForPathToExistInternal(devicePath string, maxRetries int, deviceTransport string, osStat StatFunc, filepathGlob GlobFunc) bool { for i := 0; i < maxRetries; i++ { var err error if deviceTransport == "tcp" { _, err = osStat(devicePath) } else { fpath, _ := filepathGlob(devicePath) if fpath == nil { err = os.ErrNotExist </del> <ins> func waitForPathToExistInternal(devicePath *string, maxRetries int, deviceTransport string, osStat StatFunc, filepathGlob GlobFunc) bool { if devicePath != nil { for i := 0; i < maxRetries; i++ { var err error if deviceTransport == "tcp" { _, err = osStat(*devicePath) } else { fpath, _ := filepathGlob(*devicePath) if fpath == nil { err = os.ErrNotExist } else { // There might be a case that fpath contains multiple device paths if // multiple PCI devices connect to same iscsi target. We handle this // case at subsequent logic. Pick up only first path here. *devicePath = fpath[0] } </ins> } <ins> if err == nil { return true } if err != nil && !os.IsNotExist(err) { return false } if i == maxRetries-1 { break } time.Sleep(time.Second) </ins> } <del> if err == nil { return true } if err != nil && !os.IsNotExist(err) { return false } if i == maxRetries-1 { break } time.Sleep(time.Second) </del> } return false }

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-beta.1", GitCommit:"", GitTreeState:"not a git tree", BuildDate:"2017-06-08T13:08:32Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:38:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Bare-Metal, HP SE316M, 2x X5675, 144GB, iSCSI/RDMA/iSer OS: CentOS 7 Kernel: Linux at-host-18 4.9.31 SMP Wed Jun 7 22:40:40 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others: What happened: I want to mount a iSCSI volume via iSER (RDMA over Infiniband) in this case the iSCSI iface is "iser" and not tcp, therefore the glob path for device mount is selected (), which then is passed to the fsck and mount commands without globbing the path (e.g. calling filepath.Glob) Therefore the iSCSI volume is not mounted and the pod cannot be started. Logs: What you expected to happen: To successfully mount the volume. How to reproduce it (as minimally and precisely as possible): Setup an iSCSI via RDMA (iSER) environment without multipath and try to mount a volume for a pod. Anything else we need to know: I believe it's just needed to add the globbing at line 259 like so: devicePaths = append(devicePaths, filepath.Glob(devicePath)) As I'm not a go programmer and have no experience with it, probably somebody could create a commit which I would gladly test!
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team.

You can find the team list and label list
/sig storage
I'll take a look.
/sig storage
on a local build that fixed the issue: (e.g. it compiles)
I don't want to add filepath.Glob to that general path because that line is also executed "tcp" case. I'm considering the fix below. Pretty similar.
does that work from a go specific point? is that devicePath passed by reference? (from general c/c++/php/node/c# knowledge I would say that this would have no effect on the devicePath variable .. )
Can we suppose that filepath.Glob(devicePath) always only return one device path if transport is "tcp"?
does that work from a go specific point? is that devicePath passed by reference? (from general c/c++/php/node/c# knowledge I would say that this would have no effect on the devicePath variable .. ) Thank you for above comment. I updated the logic. Could you confirm the fix on the PR?
Pretty sure that tcp transport will always only have one path - and it will exist (there's no glob in the check path)


Question: //      data to determine if an update is required. //  5.  A new timestamped dir is created //  6.  The payload is written to the new timestamped directory <del> //  7.  Symlinks and directory for new user-visible files are created (if needed). </del> <ins> //  7.  A symlink to the new timestamped directory ..data_tmp is created that will //      become the new data directory //  8.  The new data directory symlink is renamed to the data directory; rename is atomic //  9.  Symlinks and directory for new user-visible files are created (if needed). </ins> // //      For example, consider the files: //        <target-dir>/podName

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Creating a pod with a nested subpath volume will cause the pod to fail to start on Windows containerd nodes. What you expected to happen: The pods should start normally. How to reproduce it (as minimally and precisely as possible): Create the following pod on a Windows node with containerd: apiVersion: v1 kind: Pod metadata: annotations: e2e-test-privileged-psp name: pod-subpath-test-downwardapi spec: containers: args: mounttest --filecontentinloop=/test-volume --retrytime=20 image: claudiubelu/agnhost:2.21 name: test-container-subpath-downwardapi volumeMounts: mountPath: /test-volume name: test-volume subPath: downward/podname volumes: downwardAPI: defaultMode: 420 items: fieldRef: apiVersion: v1 fieldPath: path: downward/podname name: test-volume Anything else we need to know?: This issue is causing the following tests to fail on Windows Containerd nodes: [sig-storage] Subpath Atomic writer volumes should support subpaths with downward pod [LinuxOnly] [Conformance] [sig-storage] Subpath Atomic writer volumes should support subpaths with projected pod [LinuxOnly] [Conformance] The tests have the ` tag because we did not support Windows Containerd nodes at the time, and the tests still do not work for Windows Docker nodes. After a fix for this bug is merged, these tests will have to be renamed to enable them for Windows Containerd environments but not for Windows Docker environments. /sig windows Environment: Kubernetes version (use ): v1.21.0-alpha.0.493+ Cloud provider or hardware configuration: N/A OS (e.g: ): Windows Server 1809, Containerd Kernel (e.g. ): 10.0. N/A Build Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance.

The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale


Question: now := kl.clock.Now() if utilfeature.DefaultFeatureGate.Enabled(features.NodeLease) && now.Before(kl.lastStatusReportTime.Add(kl.nodeStatusReportFrequency)) { if !podCIDRChanged && !nodeStatusHasChanged(&originalNode.Status, &node.Status) { <ins> // We must mark the volumes as ReportedInUse in volume manager's dsw even // if no changes were made to the node status (no volumes were added or removed // from the VolumesInUse list). // // The reason is that on a kubelet restart, the volume manager's dsw is // repopulated and the volume ReportedInUse is initialized to false, while the // VolumesInUse list from the Node object still contains the state from the // previous kubelet instantiation. // // Once the volumes are added to the dsw, the ReportedInUse field needs to be // synced from the VolumesInUse list in the Node.Status. // // The MarkVolumesAsReportedInUse() call cannot be performed in dsw directly // because it does not have access to the Node object. // This also cannot be populated on node status manager init because the volume // may not have been added to dsw at that time. kl.volumeManager.MarkVolumesAsReportedInUse(node.Status.VolumesInUse) </ins> return nil } }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] [sig-storage] CSI Volumes [Driver: ][Serial] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow] Since when has it been failing: 2019-01-15 (when they were ) Testgrid link: Reason for failure: They all seem to be failing with: The failures seem to start with .
/sig storage
Summoning and This seems to have been failing since it was in master-blocking.
This is the same as
Thanks closing this one. /close
Closing this issue. <detailsIn response to : Thanks closing this one. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen I took a look at the latest test failures and I think it is a different issue than
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking at one test failure here: Sequence of events: stopped, pod deleted: started again: volume to dsw started and failed because it's not in the node's volumesinuse list. This repeats for a while. also started, but we skipped cleaning up because the volume is in dsw so we want to let volume manager gracefully cleanup don't remove volume from dsw because actual state has not been updated? So it seems like there's a problem with populating the node's volumesinuse after a kubelet restart.
cc
/assign will continue investigating.

i will try repro-ing at without different commits to try to figure out which pr introduced the regression
/priority important-soon
I ran these tests against a cluster with the NodeLease feature disabled, and the tests pass. So I think the issue is related to the new NodeLease feature enabled a month ago: cc
I discussed with VolumesInUse is updated periodically in the Node.Status. So we need to make sure this flow is working well with the NodeLease feature.
and I walked through the code. The NodeLease feature introduced an early exit to the node status update function, so our theory is we need to add a call to . I will make the change and test it out


Question: package validation import ( <ins> "strings" </ins> "k8s.io/apimachinery/pkg/util/validation/field" apivalidation "k8s.io/kubernetes/pkg/apis/core/validation" "k8s.io/kubernetes/pkg/apis/scheduling" ) <del> // ValidatePriorityClassName can be used to check whether the given priority // class name is valid. var ValidatePriorityClassName = apivalidation.NameIsDNSSubdomain </del> <ins> // ValidatePriorityClassName checks whether the given priority class name is valid. func ValidatePriorityClassName(name string, prefix bool) []string { var allErrs []string if strings.HasPrefix(name, scheduling.SystemPriorityClassPrefix) { allErrs = append(allErrs, "priority class names with '"+scheduling.SystemPriorityClassPrefix+"' prefix are reserved for system use only") } allErrs = append(allErrs, apivalidation.NameIsDNSSubdomain(name, prefix)...) return allErrs } </ins> // ValidatePriorityClass tests whether required fields in the PriorityClass are // set correctly.

In a recent discussion on , suggested to keep priority class names that start with "system-" for system use only and not allow user defined priority classes to start with this prefix. Given that we have reserved priority values 1 Billion for system use only, it makes sense to reserve a prefix as well to ensure that predefined priority classes won't conflict with user defined ones in the future. Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig scheduling /assign ref


Question: } } <del> if reflect.DeepEqual(currentEndpoints.Subsets, subsets) && reflect.DeepEqual(currentEndpoints.Labels, service.Labels) { </del> <ins> createEndpoints := len(currentEndpoints.ResourceVersion) == 0 if !createEndpoints && apiequality.Semantic.DeepEqual(currentEndpoints.Subsets, subsets) && apiequality.Semantic.DeepEqual(currentEndpoints.Labels, service.Labels) { </ins> glog.V(5).Infof("endpoints are equal for %s/%s, skipping update", service.Namespace, service.Name) return nil }

Currently, endpoints objects containing no subsets are decoded by the go client as subsets:[] (when requested individually) or as subsets:null (when requested in a list of endpoints). Because the endpoints controller is fed via a lister/watcher, it gets the subsets:null version fed to it. The subsets computation then returns an empty slice, which fails reflect.DeepEqual, which triggers a write attempt
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


Question: if obj := event.Object.(*api.Pod); event.Type != watch.Added || obj.ResourceVersion != "4" { t.Errorf("unexpected event: %v", event) } <del> case <-time.After(time.Millisecond * 100): </del> <ins> case <-time.After(util.ForeverTestTimeout): </ins> t.Errorf("timed out waiting for an event") } // Emit a new event and check if it is observed by the watcher.

fwiw, the time outs are way to short (100ms) please use util.ForeverTestTimeout instead.
Yes - it seems to be cause by too short timeouts. Sent out for review.


Question: func (c *iscsiDiskCleaner) TearDownAt(dir string) error { return diskTearDown(c.manager, *c, dir, c.mounter) } <ins> func portalBuilder(portal string) string { if !strings.Contains(portal, ":") { portal = portal + ":3260" } return portal } </ins>

Documentation says that port number for is optional, but I've found that not to be true. It's due to the way it detects whether the disk is attached or not. also must be an IP. Refer to:
/cc
I see what you meant. Good catch. According to , the disk path contains both ip and port. So port is indeed required even for default port 3260. If you don't have a pr yet, I'll provide a fix to the doc.
I don't have a CLA in file yet, so I'll leave it up to you. Thanks.
check this out


Question: <del> #!bash # # bash completion file for core kubecfg commands # # This script provides completion of non replication controller options # # To enable the completions either: #  - place this file in /etc/bash_completion.d #  or #  - copy this file and add the line below to your .bashrc after #    bash completion features are loaded #     . kubecfg # # Note: # Currently, the completions will not work if the apiserver daemon is not # running on localhost on the standard port 8080 __contains_word () { local w word=$1; shift for w in "$@"; do [[ $w = "$word" ]] && return done return 1 } # This should be provided by the bash-completions, but give a really simple # stoopid version just in case.  It works most of the time. if ! declare -F _get_comp_words_by_ref >/dev/null 2>&1; then _get_comp_words_by_ref () { while [ $# -gt 0 ]; do case "$1" in cur) cur=${COMP_WORDS[COMP_CWORD]} ;; prev) prev=${COMP_WORDS[COMP_CWORD-1]} ;; words) words=("${COMP_WORDS[@]}") ;; cword) cword=$COMP_CWORD ;; -n) shift # we don't handle excludes ;; esac shift done } fi __has_service() { local i for ((i=0; i < cword; i++)); do local word=${words[i]} # strip everything after a / so things like pods/[id] match word=${word%%/*} if __contains_word "${word}" "${services[@]}" && ! __contains_word "${words[i-1]}" "${opts[@]}"; then return 0 fi done return 1 } # call kubecfg list $1, # exclude blank lines # skip the header stuff kubecfg prints on the first 2 lines # append $1/ to the first column and use that in compgen __kubecfg_parse_list() { local kubecfg_output if kubecfg_output=$(kubecfg list "$1" 2>/dev/null); then out=($(echo "${kubecfg_output}" | awk -v prefix="$1" '/^$/ {next} NR > 2 {print prefix"/"$1}')) COMPREPLY=( $( compgen -W "${out[*]}" -- "$cur" ) ) fi } _kubecfg_specific_service_match() { case "$cur" in pods/*) __kubecfg_parse_list pods ;; minions/*) __kubecfg_parse_list minions ;; replicationControllers/*) __kubecfg_parse_list replicationControllers ;; services/*) __kubecfg_parse_list services ;; *) if __has_service; then return 0 fi compopt -o nospace COMPREPLY=( $( compgen -S / -W "${services[*]}" -- "$cur" ) ) ;; esac } _kubecfg_service_match() { if __has_service; then return 0 fi COMPREPLY=( $( compgen -W "${services[*]}" -- "$cur" ) ) } _kubecfg() { local opts=( -h -c ) local create_services=(pods replicationControllers services) local update_services=(replicationControllers) local all_services=(pods replicationControllers services minions) local services=("${all_services[@]}") local json_commands=(create update) local all_commands=(create update get list delete stop rm rollingupdate resize) local commands=("${all_commands[@]}") COMPREPLY=() local command local cur prev words cword _get_comp_words_by_ref -n : cur prev words cword if __contains_word "$prev" "${opts[@]}"; then case $prev in -c) _filedir '@(json|yml|yaml)' return 0 ;; -h) return 0 ;; esac fi if [[ "$cur" = -* ]]; then COMPREPLY=( $(compgen -W "${opts[*]}" -- "$cur") ) return 0 fi # if you passed -c, you are limited to create or update if __contains_word "-c" "${words[@]}"; then services=("${create_services[@]}" "${update_services[@]}") commands=("${json_commands[@]}") fi # figure out which command they are running, remembering that arguments to # options don't count as the command!  So a hostname named 'create' won't # trip things up local i for ((i=0; i < cword; i++)); do if __contains_word "${words[i]}" "${commands[@]}" && ! __contains_word "${words[i-1]}" "${opts[@]}"; then command=${words[i]} break fi done # tell the list of possible commands if [[ -z ${command} ]]; then COMPREPLY=( $( compgen -W "${commands[*]}" -- "$cur" ) ) return 0 fi # remove services which you can't update given your command if [[ ${command} == "create" ]]; then services=("${create_services[@]}") elif [[ ${command} == "update" ]]; then services=("${update_services[@]}") fi case $command in create | list) _kubecfg_service_match ;; update | get | delete) _kubecfg_specific_service_match ;; *) ;; esac return 0 } complete -F _kubecfg kubecfg # ex: ts=4 sw=4 et filetype=sh </del>

This issue is a bucket placeholder for collaborating on the "Known Issues" additions for the 1.22 Release Notes(). If you know of issues or API changes that are going out in 1.22, please comment here so that we can coordinate incorporating information about these changes in the Release Notes. /assign /sig release /milestone v1.22
GitHub didn't allow me to assign the following users: simcard0000. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This issue is a bucket placeholder for collaborating on the "Known Issues" additions for the 1.22 Release Notes(). If you know of issues or API changes that are going out in 1.19, please comment here so that we can coordinate incorporating information about these changes in the Release Notes. /sig release /milestone v1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
1.19 or 1.22 ?
/assign
For the 1.21 example in , should be changed to
Please add the following to the Known Issues section of the 1.22 Release Notes: CC
Can you also mention the CSIMigrationVSphere feature gate in the note as well?
what should I mention about the CSIMigrationVSphere feature gate? If someone uses vSphere CSI Driver, he/she should not upgrade to v1.22 whether CSIMigrationVSphere is on or off. If someone does not use vSphere CSI Driver, then we also don't need to worry about this feature gate.
If the feature gate is enabled, they should also hold off on upgrading to 1.22
The main concern as far as k8s release is - it is intree to CSI migration of vSphere volumes that is broken in 1.22. So I think our release note should say something like:


Question: <ins> kind: Service apiVersion: v1 metadata: name: dns-backend spec: ports: - port: 8000 selector: name: dns-backend </ins>

What happened: We've been doing some release readiness and auditing all tagged ginkgo tests... Looks like The test seems to expect a GOPATH when running. Im seeing some failures in sonobuoy and suspect a possible relation there but havent dove into it. Either way, its clear this test might need some updating/fixing/cleanuping and parts can be rewritten/improved. What you expected to happen: No reliance on GOPATH for an e2e test ... I dont remember the whole history on this, but no tests that rely on static files currently look in gopath, from what i can tell ? instead we plumbed in a repo-root/ option a few years ago. There may be a state of the art in terms of how we do this.... but i thought it was at one point which was an option passed in. How to reproduce it (as minimally and precisely as possible): Try to run this in sonobuoy or any other environment where there's no GOPATH or non-standard source trees, pretty sure it might not work. Anything else we need to know?: I think the fix here is to just embed the YAML in golang like we do in other tests. along the way there's other cleanup we can do to improve this test. Environment: Kubernetes version (use ): master
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
the YAML files are pretty small, so +1 to embed them in i don't think testing the YAML from the separate repository () directly is required here. ref /help /kind cleanup /priority backlog
/remove-help
/assign Also I should say, I talked to Jay on slack and he is ok with me taking a look at it, if thats ok.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle stale
/remove-lifecycle stale
xref: go 1.16 has built in support for embedding go 1.17 looks to be removing GOPATH
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/help wanted
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten still a problem. FYI re: GOPATH removal
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/assign


Question: tags = ["automanaged"], visibility = ["//visibility:public"], ) <ins> go_test( name = "go_default_test", srcs = ["sample_and_watermark_test.go"], embed = [":go_default_library"], deps = [ "//staging/src/k8s.io/apimachinery/pkg/util/clock:go_default_library", "//staging/src/k8s.io/component-base/metrics:go_default_library", "//staging/src/k8s.io/component-base/metrics/legacyregistry:go_default_library", "//vendor/k8s.io/klog/v2:go_default_library", ], ) </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Better testing of monotonic time in Why is this needed:
/sig api-machinery
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted </details
/triage accepted


Question: AdvertiseAddress net.IP CorsAllowedOriginList       []string <ins> HSTSDirectives              []string </ins> ExternalHost                string MaxRequestsInFlight         int MaxMutatingRequestsInFlight int

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add the "Strict-Transport-Security" header to the API server responses. Why is this needed: Nessus scanner treats this as a Medium Risk in kubernetes 1.18.10.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/remove-sig api-machinery /sig auth


Question: // so we only look at stdout. command := []string{ "compute", resource, "list", <del> fmt.Sprintf("--regex=%v", regex), </del> <ins> fmt.Sprintf("--regexp=%v", regex), </ins> fmt.Sprintf("--project=%v", project), "-q", "--format=json", } output, err := exec.Command("gcloud", command...).Output() if err != nil { errCode := -1 <ins> errMsg := "" </ins> if exitErr, ok := err.(utilexec.ExitError); ok { errCode = exitErr.ExitStatus() <ins> errMsg = exitErr.Error() if osExitErr, ok := err.(*exec.ExitError); ok { errMsg = fmt.Sprintf("%v, stderr %v", errMsg, string(osExitErr.Stderr)) } </ins> } <del> framework.Logf("Error running gcloud command 'gcloud %s': err: %v, output: %v, status: %d", strings.Join(command, " "), err, string(output), errCode) </del> <ins> framework.Logf("Error running gcloud command 'gcloud %s': err: %v, output: %v, status: %d, msg: %v", strings.Join(command, " "), err, string(output), errCode, errMsg) </ins> } if err := json.Unmarshal([]byte(output), out); err != nil { framework.Logf("Error unmarshalling gcloud output for %v: %v, output: %v", resource, err, string(output))

https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Loadbalancing: L7

[Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow]

[Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
the gke ingress tests started failing with: Which seems like a gcloud versioning issue. GCE tests are still green (). Any ideas on what might've changed? Probably a good idea to rewrite the parts of our helper libs that shell out to gcloud anyway, but I'm hoping to punt on that till after 11/7 (code freeze). Anything obvious I can do now to put this out?
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e

suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: []

Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] []

GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud

create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e

suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
the alias is deprecated. use instead. I believe the GKE builds use a staging version of gcloud, while GCE uses the official release. You'll probably start seeing failures on GCE next week if the underlying issue isn't fixed.

:)
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
as for your query, two things: out the in so that you can see why is failing to , since
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
thanks for the suggestion
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}
https://k8s- Failed: [] Loadbalancing: L7 [Feature:Ingress] [] GCE [Slow] [Feature: Ingress] shoud create ingress with given static-ip {Kubernetes e2e suite}


Question: import ( "errors" <ins> "fmt" </ins> "net" "reflect" "sort"

It seems like this depends on the network condition as I first noticed this while on a plane, and it happens to me every so often but never on travis. Offenders are TestSyncNodeStatusDeletePods and TestSyncNodeStatusTransitionTime. This may be reproducible by disconnecting network connection but also happens sometimes while connected. is a call to net.LookupIP(), I'll investigate more.
This also occurs on OSX over a VPN (at least, for my VPN it does). Each call to net.LookupIP(name) takes 30s and fails correctly. As a result, the entire unit test times out. That same function call fails fast on Linux.
cc
For other runtime, we can just mock out net, time. I'd really like to know how to do mocking in go. Using interface seems overkill.


Question: pdb.Status.DesiredHealthy == desiredHealthy && pdb.Status.ExpectedPods == expectedCount && pdb.Status.PodDisruptionsAllowed == disruptionsAllowed && <del> reflect.DeepEqual(pdb.Status.DisruptedPods, disruptedPods) && </del> <ins> apiequality.Semantic.DeepEqual(pdb.Status.DisruptedPods, disruptedPods) && </ins> pdb.Status.ObservedGeneration == pdb.Generation { return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are too many pdb update operations compared with other resources. ! That is, there are many "Put" operations for pdb periodically in kube-apiserver log. In my view, the amount of pdb operations is connected with disruption controller's resyncPeriod and pdb's quantity in cluster. For example: disruption-controlle's default resyncPeriod is 30s, and there are 100 pdbs in cluster. We can find that the update operation times per minute is about 200. The data is reasonable, explanation is as follows: According to disruption-controlle's design, there is one resync every 30s. One resync means to call updatePdbStatus function for every pdb, so there are 200 times updatePdbStatus. Function updatePdbStatus judges whether the pdb is changed. If changed, call apiserver to update the pdb' status. When no eviction happens, disruptedPods is empty. In this situation, disruptedPods is an empty map, but pdb.Status.DisruptedPods is nil, empty map is not equal to nil. So reflect.DeepEqual(pdb.Status.DisruptedPods, disruptedPods) returns false. The pdb is thought to has be changed, then apiserver is called to update the pdb' status. In fact, there is no change for the pdb when disruptedPods is an empty map and pdb.Status.DisruptedPods is nil. What you expected to happen: There shouldn't be so many pdb update operations. That is, when pdb has no change, apiserver shouldn't be called to update its status. How to reproduce it (as minimally and precisely as possible): Create some pdbs in cluster, you will see many "Put" operations for pdb periodically in kube-apiserver log. Anything else we need to know?: Environment: kubernetes v1.13.2 Fix suggestions: The bug is caused by the mixed usage of empty and nil for disruptedPods, the type of which is map[string]metav1.Time.

There are two fix suggestions for it: one of empty map and nil for disruptedPods, use it consistently and refuse to mix empty map and nil. For example, when disruptedPods is empty or nil, save it as an empty map in etcd. Then every client gets pdb from etcd, the pdb.Status.DisruptedPods is an empty map. get pdb for etcd, assign an empty map to it if pdb.Status.DisruptedPods is nil. Maybe the mixed usage of empty and nil for map is a common problem in k8s design, not just for disruptedPods. I will propose a work around solution for the problem. If the solution has any problem, please let me know and I can modify the code.
/sig apps
/assign Morten can you PTAL and the open PR. I reviewed as well.
ref


Question: if err != nil { return nil, fmt.Errorf("failed to parse container log max size %q: %v", maxSize, err) } <ins> // Negative number means to disable container log rotation if parsedMaxSize < 0 { return NewStubContainerLogManager(), nil } </ins> // policy LogRotatePolicy return &containerLogManager{ osInterface:    osInterface,

Setting a negative value in currently makes kubelet crash with a stack trace. The function raises an error when the value is negative, but the comment on  says that setting a negative value disables log rotation. At the moment there’s no way to disable log rotation and I believe removing lines 141 to 143 from , would allow this.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node (I guess)
/area kubelet can you show the stack trace?
The stack trace is 5000 lines long and apart from the following message, I don’t think it is relevant:
/assign
You can try to quote the value with double quotation when the value is negative, like the following: syncFrequency: 0s volumeStatsAggPeriod: 0s containerLogMaxSize: "-1"
I tried "-1", but still error
volumeStatsAggPeriod: -1s means disable
Which Kubernetes version have you installed? As if on my v1.19.4 system,no errors occured after I quoted the value with double quotation.
v1.20.2
I’m on v1.20.4. That’s already the case. Also, can only be set as a string in kubelet’s configuration, trying to set an integer will raise a parsing error (see ). The problem is not with the configuration, I’ve linked to the exact line in the code that checks for a negative value and raises the error.
Maybe you can try to disable this featureGate
Seems to work, I don’t know why I didn’t find this feature gate before.