Question: // getEtcHostsPath returns the full host-side path to a pod's generated /etc/hosts file func getEtcHostsPath(podDir string) string { <del> return path.Join(podDir, "etc-hosts") </del> <ins> hostsFilePath := path.Join(podDir, "etc-hosts") // Volume Mounts fail on Windows if it is not of the form C:/ return volumeutil.MakeAbsolutePath(runtime.GOOS, hostsFilePath) </ins> } // makeHostsMount makes the mountpoint for the hosts file that the containers

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig windows What happened: When deploying Kubernetes pods, the Linux containers will contain the following entries in the file: # Kubernetes-managed hosts file. 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters 10.240.0.67 dns-querier-1.dns-test-service.e2e-tests-dns- dns-querier-1 However, these are not set on Windows containers; the container's file does not contain any entry. This can be attributed to the fact that individual files cannot be mounted on Windows containers the same way they are on Linux containers. It can be fixed with a change to the kubelet once Windows Server 2019 has shipped, and once CRI-ContainerD is enabled on Windows. We cannot fix it for Windows Server 1803 or earlier with dockershim. What you expected to happen: The file should be managed and populated by Kubelet. How to reproduce it (as minimally and precisely as possible): Spawn a Windows pod, and check its container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cmd /S /C type C:WindowsSystem32driversetchosts Compare the result with a Linux pod's container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cat /etc/hosts Anything else we need to know?: Environment: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: N/A OS (e.g.

from /etc/os-release): Windows Server 1803 (10.0..48) Install tools: acs-engine v0.24.0 Others:
This requires single-file mapping, which won't work in Windows Server version 1803. It should be possible with Windows Server 2019 along with ContainerD
Will it work with Docker on Windows Server 2019?
Not in 18.09 or earlier. I'm not sure what release it will be in because it will require migrating to containerd.
should this issue be tracked for 1.13 in addition to If so please apply the appropriate priority and milestone labels please.
This one needs to be in v1.14. We need to get cri-containerd working first to fix this.
Thanks /milestone v1.14
By default, hosts file are not being used in Windows, it is all registry based. What features is this blocking? CC
The hosts file entries are still taken into account on Windows when it has to resolve a DNS name. This is blocking DNS conformance tests, like.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
Hello! 14 code freeze is coming in about 3 days, is this intended to be implemented in the following week?
No. It is meant to be fixed in the next release, when the Containerd support will be introduced.
/milestone v1.15
Hello! We are starting the code freeze for 1.15 tomorrow EOD. Is this issue still planned for the 1.15 cycle?
This is dependent of the ContainerD implementation. If that hasn't been implemented yet, this can't either.
Is there any update is the containerd implementation finished? Are there any relevant issues or PRs that we can use for tracking?
sounds like a no-go for 1.15, let's just move it out as no one is working on it as well. /milestone v1.16
Hello!

I'm part of the bug triage team for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16? If not please re-tag the issue to the planned milestone.
/milestone v1.17 We won't be able to do this until containerd is ready. It might work in 1.17
Bug triage for 1.17 here. This issue has been open for a significant amount of time and since it is tagged for milestone 1.17, we want to let you know that the 1.17 code freeze is coming in less than one month on Nov. 14th. Will this issue be resolved before then?
We have a PR for it, as you can see above: It;s a pretty small PR too.
The v1.17 code freeze is this Thursday, end of day Pacific time. Will this issue be closed by then?
/milestone v1.18 (code freeze v1.17)
This is going to need a Windows fix, not a ContainerD fix. We won't have that before code freeze on 5 March, so I'm pushing this out to 1.19. /milestone v1.19
This is being tracked with MS internal work item The Windows team thinks an OS fix for Windows Server 2019 will be released with the June patches. I will post updates here as I get them.
Hello from the bug triage team! This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting June 25th (about 4 weeks from now) and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for 1.19, is it still planned for this release?
can you comment on when we can expect a Windows fix for Windows Server 2019 that addresses the single file mount issues in containers when a target file already exists in the container?

we are waiting on a Windows OS fix from Microsoft to proceed here. If that fix comes in time for us to validate this functionality in 1.19 we would like it to be included in 1.19.
Hello! As this issue seems to depend on PR which has been pushed to v1.20, should this issue also be pushed to that milestone?
/milestone 1.20 The OS fix is now expect to be available as part of August Windows patches and then changes to the kubelet will need to follow. /cc
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone.

<detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.20
yes it is expected be part of August Windows release
Hey Bug Triage here. Wanted to follow up on the status of this issue as we're approaching code freeze on 12.11.2020. This issue is tagged for 1.20, is it still planned for this release?
- Yes, we'd like to make it into 1.20. We were waiting on some bugfixes in the Windows OS and those bugfixes are now available (as of Oct 20 2020).
/remove-lifecycle frozen
/priority important-soon
I believe the work involved here is to update to enable this fucntionality for WIndows if container-runtime == remote (still disable this for DockerShim) Update/Add test coverage Update documentation - possible this section ?
/milestone v1.21 Moving to 1.21 since the required OS fixes were not available in time to implement this functionality in v1.20
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hello! I am from the Bug Triage team! This issue has not been updated for a long time, so I'd like to check on the status. The code freeze is starting March 9th, 2021 (about 1 week from now). We want to ensure that each PR has a chance to be merged before code freeze. As the Issue is tagged for 1.21, is it still planned for this release?
Can this be reopened? From a related problem I have experienced using HostAlias for Windows pod deployments, this issue does not seem to be resolved. Instead, it looks as if it was closed by accident because the conversation went cold.
Hello. What Kubernetes version are you running? Also, on the Windows nodes, do you use docker, or containerd?

This is only supported on Windows nodes with containerd, since it supports single file mapping, which is required in order to mount single files into Windows containers.
We have experimented with both 1.22.11 and a 1.23 release and yea, the container runtime is containerd. Any other ideas to check up on?


Question: func (m *effectiveVersion) Set(binaryVersion, emulationVersion, minCompatibilityVersion *version.Version) { m.binaryVersion.Store(binaryVersion) <ins> m.useDefaultBuildBinaryVersion.Store(false) </ins> m.emulationVersion.Store(majorMinor(emulationVersion)) m.minCompatibilityVersion.Store(majorMinor(minCompatibilityVersion)) }

Starting in 1.31, --version no longer works to modify the build ID of the running components. /kind regression /priority important-soon Running kube-apiserver v1.31.0 with ignores the modified build ID and reports In 1.28, 1.29, and 1.30 (since ), this allowed modifying the build ID of the running server. /sig api-machinery /assign Run v1.31.0 with No response 31.0 n/a <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this broke in DefaultBuildEffectiveVersion() is grabbed in NewAPIServerCommand(), before the flag is parsed, so the GitVersion propagated into the snapshot effectiveVersion doesn't honor
cc


Question: github.com/karrick/godirwalk # github.com/konsorten/go-windows-terminal-sequences v1.0.1 => github.com/konsorten/go-windows-terminal-sequences v1.0.1 github.com/konsorten/go-windows-terminal-sequences <del> # github.com/libopenstorage/openstorage v0.0.0-20170906232338-093a0c388875 => github.com/libopenstorage/openstorage v0.0.0-20170906232338-093a0c388875 </del> <ins> # github.com/libopenstorage/openstorage v1.0.0 => github.com/libopenstorage/openstorage v1.0.0 </ins> github.com/libopenstorage/openstorage/api github.com/libopenstorage/openstorage/api/client github.com/libopenstorage/openstorage/api/client/volume

Found this when trying to see which vendored dependencies does not use a proper tag (and is using a random SHA). The dependency is very old and there have been a lot of updates to it: Looks like the However there does not seem to be anyone around to take care of the portworx plugin. Is it time to deprecate it? is anyone using it? /sig storage
cc ?
/area code-organization
Hello The portworx volume plugin is definitely in use. The portworx volume plugin doesn't need frequent updates since the plugin acts as a simple client to all provisioning calls. In the same view, the vendored openstorage package doesn't need to be updated too. The server side implementation in Portworx handles all the latest Portworx features.
cc:
in that case, can you please update the dependency to a specific tag instead of a random SHA?
/assign updated the title of the issue as well
my request is not for a new release or tag in libopenstorage/openstorage repository. Just to update the SHA we use in our k/k repository to one of the released versions ( ). This is not asking or demanding anything to be done in libopenstorage/openstorage, just that we should update what we use from there (and avoid a random SHA). We are trying to do this for all vendored libraries (one by one … slowly as part of the sig-arch code organization sub project, so it’s not a random request)
/assign
GitHub didn't allow me to assign the following users: lpabon. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After a chat with - apparently this request is coming across the wrong way I am closing this out as it seems to add a over head /close
Closing this issue. <detailsIn response to : After a chat with - apparently this request is coming across the wrong way /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: limits := v1.ResourceList{} zeroQuantity := resource.MustParse("0") isGuaranteed := true <del> for _, container := range pod.Spec.Containers { </del> <ins> allContainers := []v1.Container{} allContainers = append(allContainers, pod.Spec.Containers...) allContainers = append(allContainers, pod.Spec.InitContainers...) for _, container := range allContainers { </ins> // process requests for name, quantity := range container.Resources.Requests { if !isSupportedQoSComputeResource(name) {

What happened: A pod that has init container with request for cpu and memory, and containers that make no request for cpu or memory reports a BestEffort QoS class. This is incorrect. What you expected to happen: Containers that make requests independent of lifecycle phase should impact quality of service calculation. Found in 1.12, looks to still be present in 1.14 dev. /cc
/priority important-soon
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
I think this doesn't work very well with the current cpu manager logic. The problem is that a pod can't now have QoS class Guaranteed if it has init containers which don't have matching requests/limits. This means that normal containers in such a pod don't then get dedicated cpus even though the init containers are long gone when the app containers (with matching requests/limits) are finally started. This is quite non-intuitive. I would propose separating at least Guaranteed class calculation so that the init container resource requests wouldn't affect the calculation. What do you think?


Question: // that capture what the usage would be if the request succeeded.  It return an error if the is insufficient quota to satisfy the request func (e *quotaEvaluator) checkRequest(quotas []api.ResourceQuota, a admission.Attributes) ([]api.ResourceQuota, error) { namespace := a.GetNamespace() <del> name := a.GetName() </del> evaluators := e.registry.Evaluators() evaluator, found := evaluators[a.GetKind().GroupKind()] if !found {

the ability to get the previous version of an object directly via admission control. As a result, admission code can now use this rather than fetching the object directly. Need to update:
/cc - this is a necessary follow-up, will get to it soon.


Question: import ( "reflect" <ins> "strings" </ins> "testing" corev1 "k8s.io/api/core/v1"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-controller-manager is unable to locate the credentials of the vCenter host, even though they are defined and cased identically to the hostname in the cloud-provider-config. What you expected to happen: vCenter hostname should not be case-sensitive, or should be documented that it is. It should successfully locate credentials from the manager How to reproduce it (as minimally and precisely as possible): to an upper-case name kube-controller-manager daemon to update the secret to provision storage with the "thin" storage class will stay suck in pending Anything else we need to know?: Log Messages: ` This appears to be induced by the way the vCenter server name is calculated when the secret containing the credentials is parsed[]. The credentialKey is converted to lower case and is then parsed to lookup the server name. Environment: Kubernetes version (use ): v1.14 Cloud provider or hardware configuration: vSphere OS (e.g: ): Unknown Kernel (e.g. ): Unknown Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area/provider/vmware
/area provider/vmware


Question: // preFilterStateKey is the key in CycleState to InterPodAffinity pre-computed data. // Using the name of the plugin will likely help us avoid collisions with other plugins. preFilterStateKey = "PreFilter" + Name <ins> // ErrServiceAffinityViolated is used for CheckServiceAffinity predicate error. ErrServiceAffinityViolated = "node(s) didn't match service affinity" </ins> ) // Args holds the args that are used to configure the plugin.

Part of See example /sig scheduling /kind cleanup /help /priority important-soon
/assign /remove-help


Question: return result, f.Err } <del> func (f *FakeCloud) GetZone() (string, error) { </del> <ins> func (f *FakeCloud) GetZone() (Zone, error) { </ins> f.addCall("get-zone") return f.Zone, f.Err }

Comments that the PR bot adds solely to trigger email notification of a label change, should be immediately deleted by the bot after it adds them. I have no idea what team to assign this to... cc/
, David Oppenheimer wrote:
(Motivation, of course, is to avoid having PR threads that are 90% bot comments...)
should really be an issue over in that's where (some of) the bots live. Lets make a list of which comments we want deleted immediately. [x] Label size. [x] Needs rebase. [x] test this stale CI [x] test this pending CI [x] PR changed after LGTM, removing LGTM. [x] Comment about blocking based on the paths touched (old docs) [x] unapproved cherrypick comment [x] cherrypicks must set a milestone [x] ok to test from bot I don't want to make it all comments. There are way too many times people have asked me "Why did the bot add 'do-not-merge'?" and I can just link them the comment 2 lines above which tells exactly why....
"Test this" with an issue is also useful, isn't it? On Apr 7, 2016 6:30 PM, "Eric Paris" wrote:
OK to test Add to whitelist On Apr 7, 2016 6:30 PM, "Eric Paris" wrote:
I was envisioning deleting the by the bot. (every 48 hours after LGTM, when retesting right before merge) not those by users (with an issue) After LGTM the bot will add , that one I can remove. The request to comes from the jenkins bot. And in that I am clueless...
any interest in a review (I'm trying to not just self merge all my own bot changes). This deleted hundreds and hundreds of comments.
Looks like reviewed it since your last message:

The contains instructions for downloading kubectl. The doesn't contain these instructions but it should if it's now the recommended way of running Kubernetes locally.
+1 Just ran into this.
cc
By the way, we encourage anyone to propose improvements/fixes to docs. You can do so pretty easily by clicking the pencil icon in the top right corner of every page of
kubectl download instructions to the guide in


Question: cpuUsageCache:                  make(map[string]*cpuUsageRecord), disableAcceleratorUsageMetrics: disableAcceleratorUsageMetrics, podAndContainerStatsFromCRI:    podAndContainerStatsFromCRI, <ins> clock:                          clock.RealClock{}, </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When curling the endpoint for stats we don't see network stats. Network stats to be included when calling the summary. This works with dockershim for windows. This happens because Windows containerd uses hcsv2 to create containers and we query for the networks stats using hcsv1: This will be address eventually with move to CRI only metrics KEP: but is still several releases away. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: - name: SchedulingRequiredPodAntiAffinityWithNSSelector defaultPodTemplatePath: config/pod-anti-affinity-ns-selector.yaml <del> featureGates: PodAffinityNamespaceSelector: true </del> workloadTemplate: - opcode: createNodes countParam: $initNodes

/sig scheduling in the job is failing. success run Due to this failure, we cannot see any scheduler metrics results from the perf-dash. master
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind bug /kind flake /kind failing-test
/assign /remove-kind flake
will fixed by


Question: return nil } <del> func (fmc *FakeMirrorClient) DeleteMirrorPod(podFullName string) error { </del> <ins> // TODO (Robert Krawitz): Implement UID checking func (fmc *FakeMirrorClient) DeleteMirrorPod(podFullName string, _ *types.UID) (bool, error) { </ins> fmc.mirrorPodLock.Lock() defer fmc.mirrorPodLock.Unlock() fmc.mirrorPods.Delete(podFullName) fmc.deleteCounts[podFullName]++ <del> return nil </del> <ins> return true, nil </ins> } func (fmc *FakeMirrorClient) HasPod(podFullName string) bool {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: of a static pod (really the mirror pod) results in multiple attempts to re-create the mirror pod before it finally succeeds (note that in some runs it goes into Running or Terminating state -- it just depends upon the exact timing of the . Note that in the below trace the pod name is elided from the output. Note that the actual static pod continues to run as normal. What you expected to happen: Mirror pod should start without additional state transitions How to reproduce it (as minimally and precisely as possible): Create a static pod, it, and Anything else we need to know?: Environment: Kubernetes version (use ): 1.14/OpenShift 4.2 (master has the same code) Cloud provider or hardware configuration: AWS (not relevant) OS (e.g: ): Not relevant Kernel (e.g. ): Not relevant Install tools: Not relevant Network plugin and version (if this is a network-related bug): N/A Others:
/sig node
More details: The bug appears to be a race condition between deleting the old mirror pod, the API server deleting its record of it, and creating a replacement that's not related to skew. I suspect the reason I didn't see it bouncing back and forth in my earlier tests was simply that I wasn't running the kubectl get node frequently enough; if I run it continuously, I see it bounce. The sequence is: 1) Somebody kubectl deletes the mirror pod. 2) The static pod code in (around line 1630) notices that the pod has been marked deleted (its deletion timestamp is not nil and it is a mirror pod), deletes it (in DeleteMirrorPod in , via the apiserverClient), and creates a new one (again, via the apiserverClient). 3) The next time around, the static pod code may see either the old pod (that has been deleted) or the new one as the mirror pod.

If the old pod is the one that's seen, it sees that there's a deletion timestamp, so it deletes it again -- which, since pod deletion is by name (UID isn't involved) results in the new pod being deleted and yet another new pod being created. If the new pod is the one that's seen, then we break out with the spin cycle complete. Still TBD is why the mirror pod isn't getting replaced right away. But here are traces of the relevant sequences: Bad sequence: Good sequence: The messages at lines 7 and 18 were printed immediately after the call to apiServerClient...Create(). But the key is at lines 9 and 19. At line 9, note that the UID (...) is the UID of the old mirror pod, which has a deletion timestamp and therefore the mirror code deletes the pod -- but the pod that's actually in existence at the time is the new pod. At line 19, however, note that the mirror pod is now the new one, which is not marked for deletion, and is therefore left alone. The solution (I'll open a PR shortly) is to check both the pod name and the UID to make sure that we're in fact trying to delete the pod we think we want to delete.
Thanks for the detailed issue :) Really awesome!
I'm seeing this issue on an upgrade from v1.14.4 to v1.15.1 following the on a CentOS 7 system. After installing the updated kubelet package and restarting kubelet per the instructions, the cluster went into a loop of continually restarting the etcd pod from one control plane node and the scheduler pod from another control plane node. This continued for approximately 10 minutes before the etcd pod finally stayed up. It was another 5 minutes before the scheduler pod finally ran. I noticed v1.15.2 was released although the change log doesn't show this issue included. Was there a plan to back-port this fix to 1.15 and 1.14 (not to the same extent, but I have seen this behavior on a 1.14.4 cluster)? Since this behavior can happen to the apiserver pod, it has the potential to be very disruptive to a cluster.
The actual etcd pod, or the mirror pod (which is more likely)?

I hadn't specifically planned on backporting it, but it shouldn't be a problem; this code has been stable.
Ah, good point--the actual pod does appear to remain running. That makes it less important than I was initially thinking. However, it is a bit disconcerting for someone who happens to be looking at the state of the cluster (or for monitoring) to see the pod appear to be stuck in a restart loop. If it were just a few iterations, it might not matter too much. However, I'm seeing this happen for minutes at a time. For example, to verify that the real pod remained available I did a on the apiserver pod for one master, and output showed it in the restart loop for 15 minutes before it settled down. IMO that's worth backporting if it's low risk.
I opened a cherrypick against 1.15 for this.
Appreciated. Thanks!


Question: if config.Min != nil { floor = int(*config.Min) } <del> scaled := int(*config.MaxPerCore) * goruntime.NumCPU() </del> <ins> scaled := int(*config.MaxPerCore) * detectNumCPU() </ins> if scaled > floor { klog.V(3).Infof("getConntrackMax: using scaled conntrack-max-per-core") return scaled, nil

A new flag indicating how kube-proxy gets the number of CPUs, either from goruntime or . In kube-proxy, we can use to set node's , the resulting is . We have nodes where pods with none CPU policy and static CPU policy are running together. Pods like kube-proxy are running in none CPU policy by default, and they only gets CPUs left by static CPU policy pods. If static CPU policy pods take away most CPUs cores, kube-proxy can only get one or two cores, then will be set to a very small values. In this case, a lot packet failure can occur. When we set kube-proxy to work in static CPU policy, kube-proxy only gets the number of CPUs selected by the CPU policy, not the number of total CPUs on the node. This may surprise users who are only trying to limit kube-proxy's CPU usages, but resulting in some conntrack fulls faults. Please let me know if this enhancement is reasonable, so I can send a pr later. Thanks!
/sig network
Is it refer to deploy order of kube-proxy? If kube-proxy deploy first in node, will it get just one or two core?
By one or two I mean kube-proxy gets a small number of CPU cores, because most of CPU cores are allocated to static policy pods, sorry for the misunderstanding. The number of conntrack-max depends on the CPU cores when kube-proxy started, even kube-proxy was first deployed, it can get restarted or redeployed by some reasons. Then the conntrack-max will change.
yeah, when kube-proxy deploy firstly, Maybe it can see the whole cpu nums from shared cpu(/var/lib/kubelet/cpumanagestate when kubelet start with static cpu policy). But I think it is a bug for kube-proxy. The cpu info can also get from node resource info. So will it be suitable to add a flag to set cpu num?
Maybe, or we can add a flag for user to choose where kube-proxy should get CPU informations from. There maybe some multi-tenant scenarios where kube-proxy are deployed for per tenant so they would like to limit conntrack-max by CPU cores allocated to each tenant.
I think the simple fix is to replace with something that counts /proc/cpuinfo. I don't know of anyone running multiple proxies, and I don't think it would work well. The intention was "all the CPUs on the node". PRs welcome!
/triage accepted


Question: eventHandler(watchEvent.Object) } }() <del> return watcher.Stop </del> <ins> return watcher.Stop, nil </ins> } <del> func (e *eventBroadcasterImpl) startRecordingEvents(stopCh <-chan struct{}) { </del> <ins> func (e *eventBroadcasterImpl) startRecordingEvents(stopCh <-chan struct{}) error { </ins> eventHandler := func(obj runtime.Object) { event, ok := obj.(*eventsv1.Event) if !ok {

Follow-up cleanup work for: Most of the information about this issue can be found in , where the work merged with a to change the function signature of to return a func() as well as an error. This issue will help to track that. All TODOs introduced in are resolved. N/A N/A <detailsN/A </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority low Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /triage accepted
/priority important-longterm
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: // Exhaustively test all combinations other than trivial true->true and // false->false cases. { <del> short:       true, </del> featureGate: true, create:      step{flag: false, wantActive: 2}, update:      step{flag: true, wantActive: 0, wantStatus: v1.ConditionTrue, wantReason: "Suspended"},

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-integration-master I can't seem to find a match for beyond that? Most likely related to as it was merged during that set of commits Example Spyglass link: /cc /sig apps /priority critical-urgent /milestone v1.21
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
FYI: the error message is a red herring and isn't the real reason why this test is failing. When the integration tests are run with instead of , the mutation detector is also started, which catches a bug in the job controller where we modify the shared cache. This root cause isn't present in the logs, but you can reproduce this bug with the following command: Then apply the job controller patch in and re-run the test, you'll see that the test passes. You'll also notice that passes with/without the bug fix patch when run with . cc
Test is back on green. Thanks !


Question: # To copy an image between projects: # `gcloud compute --project <to-project> disks create <image name> --image=https://www.googleapis.com/compute/v1/projects/<from-project>/global/images/<image-name>` # `gcloud compute --project <to-project> images create <image-name> --source-disk=<image-name>` <del> GCE_IMAGES=e2e-node-ubuntu-trusty-docker10-image,e2e-node-ubuntu-trusty-docker9-image,e2e-node-coreos-stable20160622-image,e2e-node-containervm-v20160321-image </del> <ins> GCE_IMAGES=e2e-node-ubuntu-trusty-docker9-v1-image,e2e-node-ubuntu-trusty-docker10-v1-image,e2e-node-coreos-stable20160622-image,e2e-node-containervm-v20160321-image </ins> GCE_ZONE=us-central1-f GCE_PROJECT=kubernetes-jenkins GCE_IMAGE_PROJECT=kubernetes-node-e2e-images

Doesn't seem to haven been caused by my change:
/cc
The test only failed on coreos, and the is full of error messages. Plus the log is almost unreadable because of the extremely high verbosity. I filed for that. shows that the node has been rebooted a couple of times. And the kernel log shows This looks like . The hairpin mode is set to hairpin-veth. /cc
Yeah don't run with hairpin-mode I guess. is coreos using kubenet?
Node problem detector should detect this issue and report KernelDeadLock condition.
Yeah, but this is node e2e test, we don't have node problem detector running on node e2e cluster.
Not sure how to assign coreos issues... Assigning to people to reassign based on thread activity. Please reassign correctly!
ping!
/cc
This happens in node e2e test right? I believe node e2e test is NOT using kubenet.
Running node e2e with kubenet+hairpin-veth could prevent this.
Sorry, this is lost in github notifications.. Feel free to ping me on slack if I am not responding in a day next time :)
To run kubenet on a node, the framework needs to download and unpack the cni plugin in kubelet needs to run with and a fixed pod CIDR (since there is no node controller) I did those in , but it failed on the trusty images because is not installed. We need to either install this in the node e2e framework, or bake it in to the image. what's the best practice for installing packages on the images?
I believe updating is the best way to bake it into the image, and I also think that should be preferred over installing it each time. You could do the same with CNI. Thanks for working on this!
I updated the PR to install nsenter on the trusty images. As for cni, I still keep it in the node e2e framework to install it every time because 1) it only downloads from GCS and untar a file of ~14MB, 2) the tarball is maintained and updated by It'd be an extra burden to update the images just for cni updates.
OK with the plan.

Seeing the error message again after switching to kubenet+hairpin-veth do you want to take a look at the logs?
let me take a look
urrrh... we hit it again. Switching back to
There's a difference in the above kernel log; the device is not a interface but rather an interface (presumably the one within a container's net ns since there is no eth0 on the host). I think all the prior instances of this were veths on the host.
True. This one is slightly different. I think deadlock on interface also showed up before.
Haven't seen any new occurrence since landed. We'll know in a day or so whether this problem has been fixed.
AFAIK, there are no new failures after landed. It should be safe to let this issue stay closed.
unfortunately I hit the bug with hairpinmode sets to promiscuous-bridge kubelet v1.3.0, linux kernel 4.6.3 cbr0 has already enter the promiscuous mode, all veth hairpinmode = 1 start with parameter
can you ? what distro are you running?
also are you using a network-plugin, overlay (like flannel), which cloudprovider?
Are the SELinux warnings normal ?
Maybe a little out of topic. Are you running K8s on GCE? Do you have the node problem detector running in your cluster?
all veths hairpin_mode = 1 my distro is coreos 1068.6.0 I don't use any network-plugins, and no cloudprovider support here, controller-manager start with kubelet setup after address allocated by controller-manager, all pods connected each other with static routes which controlled by ospfd running inside docker. I can confirm the selinux warnings not related to the bug.
not running on GCE. no detector running inside.
Alright, just want to verify whether the NPD works as expected.
here is the logs of kubelet startup.
That's the problem, the reason we ran into this to begin with is because of hairpin mode. Apparently it is still getting set, or isn't getting erased in your environment. I don't think we erase existing values.
that's wired.

I checked all my nodes, each veth hairpin_mode is 1. it seems it was set to 1 regardless of --configure-cbr0.
there is no existing veth before kubelet create bridge , docker could not start without cbr0 which created by kubelet.
hairpin_mode is always enabled if the device is a bridge interface,
Trust me it's not on my (and our other e2e test) clusters, or we'd know because tests would fail, and teh default setup is docker running on cbr0 and using veths. In 1.3 we started using kubenet network plugin by default, but hairpin flag wasn't set since 1.2. I can trace the code path down when I have some time, but I suspect it's something to do with the way coreos is setup.
I trust you, I promise. I did some tests. None of results told me the flag is set by kubelet. I could not figure out what was happend to these veths. finally, I duplicate to , the only difference is hairepinEnable default to "0", and invoke the hairpinunset.SetUpContainerPid when false. It works! But, if I create veth by hand, the default hairpin_mode is 0 , oh my god. It will not happen in my test environment, only in production env
will enable hairpin_mode That's my fault, I should not set it to false because kubernetes do not use docker's network model.


Question: // Shutdown pod. Readiness should change to false podClient.Delete(context.Background(), podName, metav1.DeleteOptions{}) <del> err = wait.PollImmediate(framework.Poll, f.Timeouts.PodDelete, func() (bool, error) { pod, err := podClient.Get(context.Background(), podName, metav1.GetOptions{}) if err != nil { return false, err </del> <ins> err = waitForPodStatusByInformer(f.ClientSet, f.Namespace.Name, podName, f.Timeouts.PodDelete, func(pod *v1.Pod) (bool, error) { if !podutil.IsPodReady(pod) { return true, nil </ins> } <del> // verify the pod ready status has reported not ready return !podutil.IsPodReady(pod), nil </del> <ins> framework.Logf("pod %s/%s is still ready, waiting until is not ready", pod.Namespace, pod.Name) return false, nil </ins> }) framework.ExpectNoError(err) })

pull-kubernetes-e2e-kind-ipv6 Kubernetes e2e suite.[It] [sig-node] Probing container should mark readiness on pods to false and disable liveness probes while pod is in progress of terminating See also, possibly related: Kubernetes e2e suite.[It] [sig-node] Probing container should mark readiness on pods to false while pod is in progress of terminating when a pod has a readiness probe Not sure. At least as of July 27, 2022, but probably earlier than that. No response ! /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign we should use listwatch to monitor pod status changes accurately, this interval 2s may lose pod change and cause the test failure
we did the same recently in another test, didn't we? I can't find the PR
sorry, i have not committed yet. do you plan to fix this test again? if so, I can unassign
he, no sorry for the confusion, I thought you were the one that fixed it, maybe it was ? There was one e2e that was flake and was solved using a listwatch instead of polling it every few seconds, I was trying to find the PR to show you , in case it can give you some idea ... please go ahead and fix it :)
this one?
yeah, so my memory was good then, I've identified the right author :smile:
I didn't understand what you said at first, but now I understand :smile:


Question: v1core "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/client-go/pkg/version" "k8s.io/client-go/tools/record" <del> cloudprovider "k8s.io/cloud-provider" </del> <ins> "k8s.io/cloud-provider" </ins> nodehelpers "k8s.io/cloud-provider/node/helpers" servicehelpers "k8s.io/cloud-provider/service/helpers" cloudvolume "k8s.io/cloud-provider/volume"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have pretty large clusters in aws and in production, we found an inefficiency in provisioning dynamic EBS volumes, where aws-ebs volume provisioner tries to look for available zone from aws directly. With scale, this can trigger massive from ec2, and affect cluster stability. This is also related to What you expected to happen: To simply get zones with available instances, we can get information from API server for efficiency. There might be a delay between node gets shutting down and such change is reflected in api server, but in case node is not ready or gets terminated, there will be higher level reconciliation triggered so we are safe. How to reproduce it (as minimally and precisely as possible): We only expect this to be hit when the cluster is huge (or aws account is huge) so not easy to reproduce. /assign /sig aws /sig cloud-provider
/assign


Question: var _ = SIGDescribe("SchedulerPredicates [Serial]", func() { var cs clientset.Interface var nodeList *v1.NodeList <del> var totalPodCapacity int64 </del> var RCName string var ns string f := framework.NewDefaultFramework("sched-pred")

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-cos-k8sbeta-serial (ci-kubernetes-e2e-gce-cos-k8sbeta-serial) Which test(s) are failing: Various CSI Driver tests. Since when has it been failing: Earliest observed: 11/17 Consistently failing since: 11/29 Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.17 /sig storage /priority critical-urgent
/assign to investigate. are the expansion test flakes supposed to be fixed?
It appears as though some of the failures are also due to tests owned by sig-scheduling: /sig scheduling
The only failures I see in CSI+expansion is from gce-pd CSI driver. We need to fix for that.
We should move "SchedulerPredicates [Serial] validates MaxPods limit number of pods that are allowed to run [Slow]" out of e2e tests. It relies on a particular state of the cluster before the test start that is hard to validate. It should be more than enough to have an integration test for this, even a unit test (which we already hav ) should be fine .
will you be willing to work on that? it's flaking A LOT right now, definitely more than 50%
I created , should be addressed soon.
There are several tests flaky. Summarize them as below expansion test See comments from and a few other tests The cause of test failure is node is OutOfPod resource. See one of the log message {kubelet test- This is due to the SchedulerPredicates test mentioned above , which creates more than 100 Pods at the same time. We need to move this test out to make sure other tests have enough resources.
I assume this was not cherry-picked to 1.17, right?
/reopen this is still an issue on 1.17. we should probably consider cherry-picking as this is creating a lot of noise in the dashboards
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/kind flake
correct, it was not cherry-picked to 1.17.
/retitle [Flaking Test] gce-cos-k8sbeta-serial (ci-kubernetes-e2e-gce-cos-k8sbeta-serial)
/retitle [Flaky Test] gce-cos-k8sbeta-serial (ci-kubernetes-e2e-gce-cos-k8sbeta-serial)
/milestone v1.18
/cc
/assign Based on 's analysis earlier, the test failures are due to a scheduler test taking up too many resources. I believe that scheduler test fix needs to be cherry-picked. Assigning to sig-scheduler to track it. /remove-sig storage
/unassign
Those labels are not set on the issue: <detailsIn response to : /assign Based on 's analysis earlier, the test failures are due to a scheduler test taking up too many resources. I believe that scheduler test needs to be cherry-picked. Assigning to sig-scheduler to track it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Those labels are not set on the issue: <detailsIn response to : /assign Based on 's analysis earlier, the test failures are due to a scheduler test taking up too many resources. I believe that scheduler test fix needs to be cherry-picked. Assigning to sig-scheduler to track it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.17
/retitle [Failing Test] gce-cos-k8sbeta-serial (ci-kubernetes-e2e-gce-cos-k8sbeta-serial) /remove-kind flake /remove-sig scheduling /sig storage
/unassign /assign Looks like the cherry pick did not cause this to bounce back. Any ideas why that would be? cc
/milestone v1.18
Is this failure specifically for 1.17 (because we only cherry picked into 1.18)?
Looks like the 1.17 serial test grid is looking fine?

(slightly flaky)
Can we also use a new bug to track the quay related failures which are unrelated to the original issue?
This job is now green


Question: # check right pod exists kube::test::get_object_assert 'pods a' "{{${id_field:?}}}" 'a' # check wrong pod doesn't exist <del> output_message=$(! kubectl get pods b -n nsb 2>&1 "${kube_flags[@]:?}") kube::test::if_has_string "${output_message}" 'pods "b" not found' </del> <ins> kube::test::wait_object_assert 'pods -n nsb' "{{range.items}}{{${id_field:?}}}:{{end}}" '' </ins> # apply b kubectl apply -l prune-group=true -f hack/testdata/prune/b.yaml "${kube_flags[@]:?}"

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig cli /cc /milestone v1.20 /prority important-soon
after creating the CRD, before applying the CR, kubectl should wait until includes :
/assign
/assign is this issue related to ?
no, this is simply kubectl trying to apply a CR before the server has made the API available at all
Got it! and I have filed to address this issue.
Still happening:
still happening: looks like the root cause is not related to CRD: I think the message Error from server (NotFound): "myobj" not found is expected, as the CR was not actually created: /reopen
Reopened this issue. <detailsIn response to : still happening: I think the message Error from server (NotFound): "myobj" not found is expected, as the CR was not actually created: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yes, the actual error is later: Line 177 succeeds, and the next line fails:


Question: package azure_file import ( <ins> "fmt" </ins> "io/ioutil" "os" "path" <ins> "reflect" </ins> "strings" "testing"

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): azurefile azurefile 0777 Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:38:40Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Microsoft Azure Container Service OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS Kernel (e.g. ): Linux 4.4.0-28-generic -Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x8664 x8664 x8664 GNU/Linux What happened: I attempted to set up an Azure File Volume using the built-in volume type azureFile. Everything works, except that the files and directories defaults to permission 0777, always. When I tried to manually mount within the container, I could customise dirmode and filemode, meaning that this change should be possible. Reading the source code confirms it - dirmode=0777 and filemode=0777 seem hardcoded. What you expected to happen: Being able to supply dirmode and filemode to the mount. e.g. How to reproduce it (as minimally and precisely as possible): N/A. This is a feature request.

Anything else do we need to know: N/A
This is a big requirement, using an azure file volume for postgresql for instance fails due to permissions as initdb tries to change the permissions to be less restrictive but is unable to.
Evaluating compatibility between a certain setup of a volume mount and an arbitrary piece of software would be the responsibility of the integrator, wouldn't it? It seems a big requirement for K8s itself to have to ensure PostgreSQL can do what it wants. An integrator who tries to use Azure File Storage for PostgreSQL should know better to mount with the correct permission.
The point is that you cannot mount with permissions, they're hard coded to 0777.
Ah. Yeah. Sorry for misunderstanding the comment above. Let's say it is an "important" requirement :)
/sig azure
/assign andyzhangx
I have a for this, use mount options in storage class, below is an example:
Has this change made it into any release yet? I can't seem to find it... Or was the issue resolved in some other way? I'm facing the issue you originally reported, i.e. using a pod volume to an azureFile, and it always mounting with filemode and dirmode of 0755... and I need it to be 0777. Is there a workaround? TIA.
azure file mountOptions feature is available from v1.8.5, here is the example :
Thanks -- I was still having trouble until I the storageClassName: "" to the claim... then realized I was hitting issue which I can avoid by setting storageClassName: "default" to both the pv and pvc, and now everything works. Thanks again!


Question: // Updated should be called when the proxier has successfully updated the service // rules to reflect the current state. Updated() <ins> // Run starts the healthz http server and returns. Run() </ins> } <del> var _ ProxierHealthUpdater = &ProxierHealthServer{} </del> <ins> var _ ProxierHealthUpdater = &proxierHealthServer{} </ins> <del> // ProxierHealthServer returns 200 "OK" by default. It verifies that the delay between </del> <ins> // proxierHealthServer returns 200 "OK" by default. It verifies that the delay between </ins> // QueuedUpdate() calls and Updated() calls never exceeds healthTimeout. <del> type ProxierHealthServer struct { </del> <ins> type proxierHealthServer struct { </ins> listener    listener httpFactory httpServerFactory clock       clock.Clock

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: , from whatever combination of inputs we stuck into it, segfaulted. The segfault, which originates from a slightly modified fork of v1.17, looks as such: Translated to master, where the problem appears to be present, the segfault originates from this call: And that is called from here: This seems odd, because The only pointer involved in the segfaulting call is itself The caller nil-checks before calling a method on . I did some digging and found , which is a fix for this problem. However, this fix appears to be undone in . There doesn't seem to be a reason for this to have been changed in the latter PR. What you expected to happen: should not segfault from any bizarre combination of user input. If the situation is invalid, it should fail gracefully. How to reproduce it (as minimally and precisely as possible): Honestly, I'm not 100% sure what combination of inputs we crammed into , but I can locate them and post them if necessary. Anything else we need to know?: I don't feel confident opening a PR to fix this myself, but in addition to recreating the fix from , a comment should be explaining why there is this weird, seemingly pointless layer of indirection when assigning to the interface, so that this doesn't get un-fixed again.
Maybe decompose the above statement into multiple lines which should give us more idea which field was nil.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
can't be nil, as it is of type , which is not an interface or pointer is an interface. However, when built from the exported constructor, , it is set to a literal , which is likewise not a pointer Therefore, the only remaining field is itself. Plus, I don't quite recall my go segfault stack trace syntax, but I'm relatively sure that indicates that the first argument to the method is , nil, and that the first argument to methods is the object being called on.

/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
When config.HealthzBindAddress is empty, we would encounter the scenario depicted in the stack trace. I have proposed a fix.
/assign
That commit, , is part of the PR that I believe introduced the problem. I strongly believe the problem to be one of a non-nil interface with a nil receiver. This is the only way that could be nil in the snippet of code from . Before PR (the changes from which are present in the version I ran), there was this odd looking set of code: The variable healthzServer, which is a pointer of type , was assigned to the variable , which is an interface of type . Then, later on, the variable was passed to , as an argument of type . This looks unexpected, and it appears that the variable is extraneous. After the changes in PR , the variable was removed, and was passed directly to as the argument for interface type . The former set of changes, with the intermediate variable, are meant to avoid the situation of a non-nil interface with a nil value. If the variable is never set in the conditional statement, then it is a nil interface. If is passed directly, then the result is the argument to is an an interface who's value is and who's concrete type is . This means that is called with as the receiver. I strongly believe this to be the root cause of the issue, but I do not know enough about kubernetes to open a PR with the correct fix, which would be to reinstate the variable (after the renaming in , this would be a variable of type ) to use as a go-between, to once again avoid this situation of a non-nil interface with a nil receiver. I believe that the removal of the variable was out of a misunderstanding of its purpose. If my assessment of the problem is incorrect, please let me know so I can understand where my logic is wrong. It is very possible that I am misunderstanding the phenomenon of a non-nil interface with a nil receiver. I would open a PR myself, but I do not know enough about Kubernetes to be confident in my ability to correctly fix and test this problem.
/remove-triage unresolved


Question: // Creates a new event broadcaster. func NewBroadcaster() EventBroadcaster { return &eventBroadcasterImpl{ <del> Broadcaster:   watch.NewBroadcaster(maxQueuedEvents, watch.DropIfChannelFull), </del> <ins> Broadcaster:   watch.NewLongQueueBroadcaster(maxQueuedEvents, watch.DropIfChannelFull), </ins> sleepDuration: defaultSleepDuration, } } func NewBroadcasterForTests(sleepDuration time.Duration) EventBroadcaster { return &eventBroadcasterImpl{ <del> Broadcaster:   watch.NewBroadcaster(maxQueuedEvents, watch.DropIfChannelFull), </del> <ins> Broadcaster:   watch.NewLongQueueBroadcaster(maxQueuedEvents, watch.DropIfChannelFull), </ins> sleepDuration: sleepDuration, } } func NewBroadcasterWithCorrelatorOptions(options CorrelatorOptions) EventBroadcaster { return &eventBroadcasterImpl{ <del> Broadcaster:   watch.NewBroadcaster(maxQueuedEvents, watch.DropIfChannelFull), </del> <ins> Broadcaster:   watch.NewLongQueueBroadcaster(maxQueuedEvents, watch.DropIfChannelFull), </ins> sleepDuration: defaultSleepDuration, options:       options, }

The standard event recorder, when recording an event, calls the underlying Broadcaster in a goroutine, so that even recording is non-blocking. Things like leader election emit events over the course of their lifecycle via this broadcaster. Consider the following scenario: GR1: spawn leader election ----GR2 GR2: Leader election: emit event "Lease Acquired" ----spawn GR3 () GR1: Shutdown leader election ----blocks until GR2 is done GR1: Shutdown broadcaster --() GR3: ----send on closed channel () Practically, we've observed this behavior popping up occaisionally in tests in controller-runtime:
/sig api-machinery /kind bug
<details<summaryrelevant race detector output/stack traces</summary</details
were you planning on sending a fix, or need our help to send one to fix it?
I can probably try to figure out a fix if you want, but the whole model for how event broadcasters currently work is inverse to how Go suggests you manage channels (only writers should ever close channels, not readers). I'll need to do a bit of playing around with things.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think this is fixed by . Please reopen if it's not the case.
/triage accepted
Would be possible to cherry-pick this fix in 1.20?


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package azure </ins>

make verify WHAT=typecheck-providerless outputs /cc /milestone v1.19
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
see example successful run:
For reference -
fix in
screaming


Question: // considered for preemption. // We look at the node that is nominated for this pod and as long as there are // terminating pods on the node, we don't consider this for preempting more pods. <del> func podEligibleToPreemptOthers(pod *v1.Pod, nodeInfos framework.NodeInfoLister) bool { </del> <ins> func podEligibleToPreemptOthers(pod *v1.Pod, nodeInfos framework.NodeInfoLister, nominatedNodeStatus *framework.Status) bool { </ins> if pod.Spec.PreemptionPolicy != nil && *pod.Spec.PreemptionPolicy == v1.PreemptNever { klog.V(5).Infof("Pod %v/%v is not eligible for preemption because it has a preemptionPolicy of %v", pod.Namespace, pod.Name, v1.PreemptNever) return false } nomNodeName := pod.Status.NominatedNodeName if len(nomNodeName) > 0 { <ins> // If the pod's nominated node is considered as UnschedulableAndUnresolvable by the filters, // then the pod should be considered for preempting again. if nominatedNodeStatus.Code() == framework.UnschedulableAndUnresolvable { return true } </ins> if nodeInfo, _ := nodeInfos.Get(nomNodeName); nodeInfo != nil { podPriority := podutil.GetPodPriority(pod) for _, p := range nodeInfo.Pods {

What happened: When a Pod's nominated node is considered as UnschedulableAndUnresolvable in the filter stage, the scheduler doesn't consider that pod for preemption even there are pods can be preempted on another node. What you expected to happen: When the pod's nominated node is considered as 'UnschedulableAndUnresolvable ' (For example, the taint being by those conditions, ), the scheduler should try to preemption on other nodes. How to reproduce it (as minimally and precisely as possible): Create low priority pod, which schedule to nodeA Create mid priority pod, which schedule to nodeB Create high priority pod, which preempt the low priority pod on nodeA, and set high priority pod's nominated node as nodeA The nodeA's kubelet is down, the node life cycle controller taint the nodeA with '' (or taint with disk ) before low priority pod finishes the termination. The high priority pod is keeping as pending status. The scheduler doesn't try to preempt the mid-priority pod on nodeB Anything else we need to know?: Environment: Kubernetes version (use ): 18.0 Cloud provider or hardware configuration: baremetal OS (e.g: ): ubuntu1804 Kernel (e.g. ): 0-46-generic Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
/assign
This seems to be a duplicate of .
actually, I found this case after review . I file this issue since I feel it should be another case. The want to cleanup the nomnated node for cluster autoscaler. This is case need to allow the pod to preempt on other nodes again even it has a nomanted node. But I may understand incorrectly, please mark it duplicated if I'm wrong. Thanks for help anyway. :)
If the nominatedNodeName can be cleared properly in any case, this issue will no longer exist. From this point of view, I'd say this issue is a sub-case of .
BTW: in your description, if "The nodeA is down", how it comes out "a Pod's nominated node is considered as UnschedulableAndUnresolvable in the filter stage"? nodeA should NOT have existed in the result of Filter(), right?

(if the nodeA is just being shutdown, we still treat it as a normal node)
If the NodeA is down, the node life cycle controller will add taint '' to the node, then the TaintToleration filter will return 'UnschedulableAndUnresolvable '. It also means that it isn't only about nodeA is down, there are other conditions in this ' to return 'UnschedulableAndUnresolvable'. Beyond the TaintToleration filter, other filters may return 'UnschedulableAndUnresolvable' also, as my understanding, that means that node won't be available for that pod anymore, then we should try other nodes for preemption.
then we should better describe it as "being applied with a Taint", which is more general. When you say "node is down", it usually refers to the node is deleted physically.
I got it, thanks again! let me correct it
Also it's different from . Let's not mention "node is down" in the description to avoid confusion.
thanks, done :)


Question: test.cachedItem.fit, test.cachedItem.reasons, test.equivalenceHashForUpdatePredicate, <ins> true, </ins> ) }

Fix this TODO (raised by : Part of: One thoughts to fix this is, we can redesign the equivalence cache data structure to: bool instead of deleting the entry. certain version tag in the entry, update the version per WRITE op (Update and Invalidate). the version after eCache predicate, and check it before update eCache.
Another possible approach is: eCache before running a predicate. the predicate. the eCache for the predicate. eCache. Basically, in the existing code all predicates are run first and their results are collected and then at the end of the function the eCache is updated. Instead of that, we can move the eCache update for each predicate inside the for loop and next to each predicate execution and synchronize before start of the execution of each predicate. The downside to my suggested approach is that performance may suffer due to frequent locking and unlocking.
Prefer to bsalamat@ 's option which is simpler :). Comparing to performance improvement by eCache, the lock is acceptable to me.
yup same here, going with the locking is easier and is acceptable. - also using a flag for invalidation might just move the problem from eCache to the flag ?
Oh I missed this notification somehow, busy on cleaning scheduler integration test. Yep so if locking is acceptable, it should be cleanest way to fix this, optimistic control would be a more complicated story.
- are you handling this one ?
Already been taken by volunteer days ago :)


Question: HostPort:      *pm.HostPort, ContainerPort: *pm.ContainerPort, Protocol:      proto, <ins> HostIP:        pm.HostIP, </ins> }) } return portMappings, nil

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm using a proxy-pod to forward both and privileged ports to an service using the following configuration: However, when I start this service, it somehow highjacks the ports for other host IPs as well. I have a legacy nginx-proxy running on the same server, inside docker bound on specific non-conflicting IPs, but as soon as I start the pod above, it breaks SSL termination of the existing nginx proxy (outputing the "Fake ingress certificate"). Existing docker-compose: Name Command State Ports
nginxproxy nginx -g daemon off; Up 5.135.123.160:443-443/tcp,5.135.123.162:443-443/tcp, 5.135.123.160:80-80/tcp,5.135.123.162:80-80/tcp nginxproxydockergen /bin/sh -c /usr/local/bin/ ... Up nginxproxyletsencrypt /bin/bash /app/entrypoint. ... Up [...] Server certificate: subject: start date: Jun 11 21:55:37 2018 GMT expire date: Sep 9 21:55:37 2018 GMT subjectAltName: host "" matched cert's "" issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3 SSL certificate verify ok.

iptables -t nat -nvL
I've flushed/dropped all iptables then reboot and again to make sure I had a clean state. Here is the iptables snapshots along the setup: Can confirm that SSL breaks for other IPs as soon as I start the pod proxy (last step). Looks like the culprit is . Destination should be and not . : Does work if I manually patch the IPtables rules: Which gives: Is there a way to automate this kind of patching to make sure the rules are not reverted by kubernetes? Thanks for the kind help :)
Thx for the reply about the iptable rules. According to the describe from u, I guess the cni plugin u used dont use the pod's data clearly. Maybe in the cni flannel,The flannel just change the rule by the hostport. And it think all the ip packets in the host must do the dnat and snat. I think flannel dont catch the hostip in the pod's yaml.
Hi mgcrea. Maybe i know why this happen. Maybe It is not the flannel's problem. I will try to fix this. /assign liucimin
GitHub didn't allow me to assign the following users: liucimin. Note that only and repo collaborators can be assigned. <detailsIn response to : Hi mgcrea. Maybe i know why this happen. Maybe It is not the flannel's problem. I will try to fix this. /assign liucimin Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
any update on this?

Did you ever find out the root cause? /remove-lifecycle rotten
Yes,I have push a pr in k8s.
v1.11.5 also the problem. Has Version 1.12 been fixed? thanks.
The pr is not merge yet. So the problem is also in v1.12 and v1.13


Question: petMounts := []api.VolumeMount{{Name: "datadir", MountPath: "/data/"}} podMounts := []api.VolumeMount{{Name: "home", MountPath: "/home"}} ps := newStatefulSet(psName, ns, headlessSvcName, 2, petMounts, podMounts, labels) <ins> setInitializedAnnotation(ps, "false") </ins> _, err := c.Apps().StatefulSets(ns).Create(ps) Expect(err).NotTo(HaveOccurred())

What happened: What you expected to happen: The statefulset should have scaled down to 0. Instead, even after 10 minutes: How to reproduce it (as minimally and precisely as possible): Run the e2e test on a gce cluster running the latest and greatest kubernetes. cc
if the petset has replicas=0 it's not going to do anything
When I a running petset, and set its scale to 0, that seems to work and delete the pets, and we should have a functioning scaler. Am I misunderstanding something? I'm using the same teardown process for the e2e test I linked in the issue as the other tests (), but it times out sometimes after network partition.
yeah is scale in the petset controller noticing deletion timestamp? because the test code you showed doesn't care
As I mentioned yesterday, it isn't even issuing a delete request even after 10 minutes, because the pods continue to be "running". I'll check the controller manager logs today. I suspect it might be the backoff in the rate limited work-queue.
Hmm, the AGE fields on your pets don't make sense if it isn't even issuing a delete?
In the output I pasted above, at the 7 minute mark, it created the new replacement pet-0, after the network partition was healed. Then tried to scale down the StatefulSet and waited 10 minutes for the pods to disappear.
I see a bunch of different failures modes on the bug: 3 08:58:44.475: Failed waiting for pods to enter running: timed out waiting for the condition Failed to scale statefulset to 0 in 10m0s. Remaining pods: [] 3 08:11:30.728: Unexpected error: <nilBasically 3 translates to We should clarify. 1 and 2 both have events like guessing the node controlelr shouldn't be setting timestamp?
Looking into the case 2: The controller manager log goes: So I guess even deletions are being blocked by the health/readiness.
Both issues appear to be due to the testsuite and the way we use the "initialized" annotation in tests. I'll send a PR to fix these shortly. Thanks


Question: // need hardcoded defaults. const DefaultVolumeType = "gp2" <del> // Used to call RecognizeWellKnownRegions just once </del> <ins> // Used to call recognizeWellKnownRegions just once </ins> var once sync.Once // AWS implements PVLabeler.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: From /kind cleanup What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cloud-provider /assign


Question: }, }, }) <ins> if err := e2epod.WaitForPodCondition(ctx, f.ClientSet, f.Namespace.Name, podName, "Ready", 120*time.Second, testutils.PodRunningReady); err != nil { framework.ExpectNoError(err, fmt.Sprintf("Pod %v could not enter running/ready", podName)) } </ins> var ( ngPids []int err    error

Test: [sig-node] Restart [Serial] [Slow] [Disruptive] Kubelet should correctly account for terminated pods after restart I've also seen on /kind flake /sig node Looks like occasionally we have a second 50m CPU pod hanging around on the node, blocking enough pods getting scheduled for the test: I'm a bit puzzled because no pods are listed as currently running when the node dumps debug info. Could be a bug?
another case [sig-node] Container Manager Misc [Serial] Validate OOM score adjustments [NodeFeature:OOMScoreAdj] affected the Restart test case, see the kubelet log, the kubelet-container-manager guaranteede9d81584 pod was received almost simultaneously with the restart-test pod
Since at 0316 21:25:08, the container of pod guaranteede9d81584 has just been creating, but the test case [sig-node] Container Manager Misc [Serial] has already finished at 0316 21:25:06, so the test case check condition may be not strict
/triage accepted /assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: -o, --output="": Output format. One of: json|yaml|template|templatefile. --output-version="": Output the formatted object with the given version (default api-version). -l, --selector="": Selector (label query) to filter on <del> -t, --template="": Template string or path to template file to use when -o=template or -o=templatefile. </del> <ins> -t, --template="": Template string or path to template file to use when -o=template or -o=templatefile.  The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview] </ins> -w, --watch=false: After listing/getting the requested object, watch for changes. --watch-only=false: Watch for changes to the requested object(s), without listing/getting first.

periodic-conformance-main-k8s-main Kubernetes e2e suite.SynchronizedBeforeSuite 07/17 23:52PDT We saw a similar issue in , which was fixed by upgrading the control plane OS from Ubuntu 18.04 to 22.04. /sig cluster-lifecycle
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
/sig node It seems to be related to SWAP.
There is a bug in containerd that is fixed in v1.6.14. The test here is using v1.6.8. Kubernetes v1.27 supported containerd versions: 1.7.0+, 1.6.15+
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still failing after is merged. Have we tried bumping the control plane OS image to ubuntu 22.04, like ?
Nevermind, the latest testgrid job might have been triggered before the PR was merged. I am observing this and will close this once I see the job passes.
Containerd version is already v1.6.21 in 's log There maybe still other problems.
can fix a similar failure in .
/close The failure of the swap is fixed by . The current failure is a flake that is mentioned in
Closing this issue. <detailsIn response to : /close The current failure is a flake that is mentioned in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // checkExternalServiceReachability ensures service of type externalName resolves to IP address and no fake externalName is set // FQDN of kubernetes is used as externalName(for air tight platforms). func (j *TestJig) checkExternalServiceReachability(svc *v1.Service, pod *v1.Pod) error { <ins> // NOTE(claudiub): Windows does not support PQDN. svcName := fmt.Sprintf("%s.%s.svc.%s", svc.Name, svc.Namespace, framework.TestContext.ClusterDNSDomain) </ins> // Service must resolve to IP <del> cmd := fmt.Sprintf("nslookup %s", svc.Name) _, err := framework.RunHostCmd(pod.Namespace, pod.Name, cmd) if err != nil { </del> <ins> cmd := fmt.Sprintf("nslookup %s", svcName) _, stderr, err := framework.RunHostCmdWithFullOutput(pod.Namespace, pod.Name, cmd) // NOTE(claudiub): nslookup may return 0 on Windows, even though the DNS name was not found. In this case, // we can check stderr for the error. if err != nil || (framework.NodeOSDistroIs("windows") && strings.Contains(stderr, fmt.Sprintf("can't find %s", svcName))) { </ins> return fmt.Errorf("ExternalName service %q must resolve to IP", pod.Namespace+"/"+pod.Name) } return nil

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-windows jobs: Which test(s) are failing: [sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance] [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] Since when has it been failing: It was a false positive until recently. Testgrid link: Reason for failure: Partially Qualified Domain Names are not currently supported on Windows containers. The tests are checking PQDNs for validation. The test was passing until recently because was still returning the exit code 0, even if the given DNS name was not found: [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] [BeforeEach] [sig-network] Services STEP: Creating a kubernetes client Apr 28 21:24:38.587: INFO: kubeConfig: /home/ubuntu/kube/config200424 STEP: Building a namespace api object, basename services STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in services-2633 STEP: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-network] Services [It] should be able to change the type from NodePort to ExternalName [Conformance] STEP: creating a service nodeport-service with the type=NodePort in namespace services-2633 STEP: Creating active service to test reachability when its FQDN is referred as externalName for another service STEP: creating service externalsvc in namespace services-2633 STEP: creating replication controller externalsvc in namespace services-2633 I0428 21:24:42. ] Created replication controller with name: externalsvc, namespace: services-2633, replica count: 2 I0428 21:24:45. ] externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:48. ]

externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:51. ] externalsvc Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:54. ]

externalsvc Pods: 2 out of 2 created, 2 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady STEP: changing the NodePort service to type=ExternalName Apr 28 21:24:55.544: INFO: Creating new exec pod Apr 28 21:25:06.371: INFO: Running '/home/ubuntu/rune2e/kubernetes/output/dockerized/bin/linux/amd64/kubectl --server=https://kubetest--83ef-11ea-ae60- --kubeconfig=/home/ubuntu/kube/config200424 exec --namespace=services-2633 agnhost-pod -- /bin/sh -x -c nslookup nodeport-service' Apr 28 21:25:09.610: INFO: stderr: "+ nslookup nodeport-servicen kube- can't find nodeport-service: Non-existent domainrrn" Apr 28 21:25:09.610: INFO: stdout: "Server: kube-rnAddress: 10.0.0.10rnrn" STEP: deleting ReplicationController externalsvc in namespace services-2633, will wait for the garbage collector to delete the pods Apr 28 21:25:10.429: INFO: Deleting ReplicationController externalsvc took: 423.628373ms Apr 28 21:25:10.830: INFO: Terminating ReplicationController externalsvc pods took: 400.686463ms Apr 28 21:25:15.792: INFO: Cleaning up the NodePort to ExternalName test service [AfterEach] [sig-network] Services Apr 28 21:25:16.230: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready STEP: Destroying namespace "services-2633" for this suite. [AfterEach] [sig-network] Services â¢{"msg":"PASSED [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance]","total":2,"completed":2,"skipped":2913,"failed":0} The issue was observed with an updated, which doesn't return 0 if the given DNS name was not found. Anything else we need to know*: /sig testing /sig windows


Question: tags = ["automanaged"], deps = ["//vendor:k8s.io/apimachinery/pkg/api/testing"], ) <ins> go_test( name = "k8s.io/apimachinery/pkg/api/testing_test", srcs = ["k8s.io/apimachinery/pkg/api/testing/valuefuzz_test.go"], library = ":k8s.io/apimachinery/pkg/api/testing", tags = ["automanaged"], ) </ins>

The apimachinery roundtripping test does not ensure that deep copies work. We need to provide a deep copy test that makes sure we don't end up with shallow copies.
I looked into gofuzz some days ago and noticed that it does not support value-only fuzzing that preserves object structure (= does not change pointer values). So we need something clever, maybe reflection based here. Any other idea?


Question: { Key:      "security", Operator: metav1.LabelSelectorOpDoesNotExist, <del> Values:   []string{"securityscan"}, </del> }, }, },

A significant number of the of are commented out, possibly was done by mistake in the past. I uncommented the tests to see if they pass, but one of them failed. What we need is to uncomment those tests, fix the one that is not passing and remove the redundant ones if there are any. /sig scheduling /help
/assign I will be moving some of the tests to unit tests and leave a cleaner set of integration tests.
/remove-help
would you like to take this? I can provide guidance.
/help /unassign
/assign
/remove-help


Question: "strings" "time" <ins> "golang.org/x/sync/singleflight" </ins> corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/api/resource"

Recently we observed high latencies on POST pods on our scalability tests. For more context see: It appeared to be throttling related, e.g.: As an example look at resource quota admission controller. In there was some fallback to use lru cache in case of on objects. However, not what happens if we are sending multiple POST pods at the same time to apiserver. What happens in Admit() if there are no results in Indexer, we list ResourceQuotas. But if there are multiple calls at the same time, there can be multiple LISTs before the cache is updated: But this can cause throttling of these requests. What we should do is: if there is List() in flight, we should wait until it is finished and lookupCache is updated.
- thanks for the analysis. - fyi, this is an interesting side-effect of the LRU cache pattern, not sure if we have adopted it in other places as well, but good to keep in mind.
To clarify, it was adopted in and admission plug-ins in separate PRs.
- it took me some time, but was useful experience. Thanks for assigning yourself.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ${REBOOT_SKIP_TESTS[@]:+${REBOOT_SKIP_TESTS[@]}}  ${GCE_FLAKY_TESTS[@]:+${GCE_FLAKY_TESTS[@]}}  ${GCE_PARALLEL_SKIP_TESTS[@]:+${GCE_PARALLEL_SKIP_TESTS[@]}}  <ins> ${GCE_SLOW_TESTS[@]:+${GCE_SLOW_TESTS[@]}}  </ins> )"} ;;

For example: I'm disabling the test in But we need to re-enable it, or roll-back the test.
fyi
Shouldn't this is part slow test, and the fix should be either: 1) Increase Kubernetes-e2-gke-ci timeout value if decide to include slow to that project or 2) disable all slow tests on that project? cc/ Which one is required here?
The test runs 50 pods per node, which is over the default --max-pod limit of kubelet. The test relies on the flag --max-pods=100 being set in cluster/gce/config- I will disable it for all GKE tests in On the other hand, it's not clear to me where the timeout is set for a specific build. /cc
Timeouts for each Jenkins build are in the Jenkins project configurations themselves, which are not currently saved in version control anywhere. :
Discussed with about the test offline. There are several issues: Do we want to run any kind of kubelet resource monitoring test on GKE? Is it useful? We only override max-pods in GCE. Should the test itself figure out what platform it can run based on the #pods in the test (see ) Per-test configuration allows greater flexibility (e.g., we may run 100 pods on GCE to test kubelet's scalability), but it'd make the tests more complicated as they need to be aware of the test environment (which many e2e tests already do); there is also the burden of updating the tests once the max-pod value changes. Does this test belong to SLOW tests or should it belong to a new category? Not all slow tests have the same attributes. Arguably this test could be enabled only on selective builds. The downside would be more flags -- less readable/manageable e2e configs. Not all builds which run SLOW tests have the proper timeout value to run this test. We need to manually change them every time a slow test is . Re-assigning this back to since she has more concerns about the PR.
Had offline discussion with We are going to split current Kubelet resource usage tracking test into two, one is targeted for GCE, another one is for GKE. In this case, we won't mess up the test code itself by introducing knobs. Re-assign this back to