Question: func TestPrivilegeContainerDisallowed(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) <ins> testKubelet.fakeCadvisor.On("VersionInfo").Return(&cadvisorapi.VersionInfo{}, nil) testKubelet.fakeCadvisor.On("MachineInfo").Return(&cadvisorapi.MachineInfo{}, nil) testKubelet.fakeCadvisor.On("ImagesFsInfo").Return(cadvisorapiv2.FsInfo{}, nil) testKubelet.fakeCadvisor.On("RootFsInfo").Return(cadvisorapiv2.FsInfo{}, nil) </ins> kubelet := testKubelet.kubelet capabilities.SetForTests(capabilities.Capabilities{

Currently we query the apiserver for the node (indirectly: ) before populating hostip on a pod. I believe this is because the node controller is responsible for calling into the cloud provider and populating the ip address on a node: This is disappointing for 2 reasons: makes using ips to identify nodes harder (). 't separate concerns sufficiently. If the reflector fails to watch nodes reliably pods won't get a hostip.
Also see .
I would argue that if we're unable to watch nodes reliably, we have many other problems. That seems like something we need to fix, rather than work around.
watch is watch, a separation of concerns between the kubelet (which is running on the node that has the hostip) and the apiserver is a different problem imo.
After discussion, we decided to just remove host IP from PodStatus. (It will still be available from "kubectl get nodes" if someone needs it.) does that sound OK with you?
I'm working on making the nodes register themselves rather than having the node controller do it for them (see which should turn into a PR later this week). Will that assuage your concerns here?
I hadn't looked at your PR. We discussed having Kubelet pull ths information from the cloud provider but thought it might not be a good idea due to (1) requiring Kubelet to understand the details of all the cloud providers, and (2) making very hard to rate limit the calls into the cloud provider. I could imagine an approach where we still use NodeController to get the node address, but the node still initiates the registration into the cluster. Not sure whether that's a good idea, though.
Kubelet won't necessarily (and really shouldn't) have the credentials necessary to contact the cloud provider. I don't remember why HostIP was to PodStatus, and I would favor removing it, except that it would be a breaking change. You could ask on google-containers@ whether anyone is relying upon it. HostIP is tricky, as hosts may have multiple addresses, in general, which is why NodeAddresses is now an array.
On GCE / AWS do you even need credentials to determine your external IP? I thought you got it from the metadata server running locally without needing to hit an Auth'd API endpoint.
HostIP wasn't the external IP AFAIK.

It's not well defined what it is, which is one reason it should be eliminated. Yes, one should be able to get the host IP from metadata, but the cloudprovider library doesn't currently do that.
I'm also in favor of removing HostIP from pods. But I don't agree with the reason in the OP (that sometimes it'll be blank when kubelet hasn't heard from apiserver)-- I think that's par for the course.
Should we schedule a meeting to discuss whether it's reasonable for Kubelets to pull their IPs directly from the cloud provider? What is the alternative -- a new API endpoint on the master where unregistered nodes can query their IP address from NodeController? Do nodes really need their IP address when they register with the master?
That sounds like a good idea (I think we can likely clear this up with a quick meeting rather than going back and forth for days in github comments).
We all agreed that we should remove HostIP from pods. Nodes do not need to provide their address(es) when they register. It's not part of the spec. They could provide whatever hostnames and/or addresses they can get through normal Linux mechanisms and/or could contact a local metadata server, but I don't think that's necessary, since the node controller will update the addresses once the node is registered, unless we want to support scenarios with no cloud provider and no external node database. In any case, we shouldn't discuss self-registration in this issue. There are other issues that cover that topic.
OK, so this issue just boils down to removing HostIP from PodStatus, and we'll discuss the rest in ?
Ping on this. e2e "Pods should get a host IP [Conformance]" is currently disabled everywhere. Bumping this to and .
Today Kubelet fills PodStatus.HostIP in generatePodStatus() in Master (e.g. NodeController) is not involved. Currently the test waits two minutes for a pod to get a HostIP Considering that we have proposed to remove this field completely, I think we should just bump up the timeout a bit and see if that fixes the test. Any other thoughts? I have also sent out email to the appropriate mailing lists to see if anyone is using PodStatus.HostIP, since we would like to get rid of it, as discussed upthread.

cc/
BTW the title of this issue says "Kubelet shouldn't require apiserver to populate host ip of pods." But it wasn't clear to me where the dependency on the API server is (unfortunately 's original comment points to a very old revision to so I'm not sure which line he was referring to). The work of finding the HostIP seems to be done by Kubelet.GetHostIP() / Kubelet.GetNode() which assumes Kubelet.nodeInfo is filled in -- is that the dependency? I couldn't find where that gets filled in. would be great if you could shed some light.
I haven't looked at this code recently but nodinfo appears to still be retrieved from a cache populated via apiserver watch:
Ah yes, that looks right, thanks (well, "right" in the sense of "that's where it gets populated," not in the sense of "it makes sense that kubelet is using code defined in the scheduler predicates for this"...) while you're here, do you have a suggestion of the best course of action? Long-term we want to get rid of PodStatus.HostIP, but short-term we need to do something about this test. How can we make populating the cache more reliable? Or should we just increase the timeout on the test? Or something else?
It looks like the kubelet already has the node address: , so assuming no one else is going to change the ip, we could just store it in the kubelet and add it for every pod? Unsure if this is a safe assumption.
I also haven't debugged this new set of test failures, so I'm not sure if it's failing for the same reason
... to make on all clouds
There isn't a new set of failures - Ike just noticed that the test is disabled, from when this issue was originally opened. could one of you comment on whether Prashanth's suggestion sounds reasonable?
I didn't parse through the entire issue since a lot of information is stale. Yes David is right, we moved to the logic of filling hostIP of Pod from apiserver to kubelet long time ago. Recording to the comment of Prashanth: Some users might have node with multiple network interfaces, and want to hostIP with specific one by updating Node object.
Fwiw I think we should turn the test on and handle things as they come up.

I think the major source of flake was handled when we did:
With respect to the long-term future of HostIP: I'm in favor of documenting the field as deprecated and asking users to contact us with their use case if they need it (in the API field documentation). As long as the field exists, I agree we should test that it's set correctly.
in this comment you wrote "Ping on this. e2e "Pods should get a host IP [Conformance]" is currently disabled everywhere. Bumping this to P1 and flake." What made you say it is disabled? I looked at Jenkins and it seems to be running it: (root)/Kubernetes%%20suite/PodsshouldgetahostIPConformance/history/ I looked in and didn't see any special handling of [Conformance]. There is an env var CONFORMANCETESTSKIP_REGEX in hack/gingko- but it doesn't seem to be set anywhere. So I don't see anything that should be causing this test to not be run. (As always there's a high likelihood I'm misunderstanding something.)
Sorry, I'm totally blind, the Jenkins history I linked to clearly shows it is being skipped. But I'm still not sure what is causing it to be skipped.
Sorry I wasn't clear; it's being skipped because it was marked pending via . Let's agree to use and labels in the future for clarity's sake.
Also see for use cases of "I need to know my local node" and possible fixes.
I'm going to remove the flake label, since has been merged and that was the suggestion here We can re-add the label if it flakes again. I'm not closing the issue, because we want to deprecate the field longer-term.


Question: ) // create test pod data structure <del> pods := newTestPods(testArg.podsNr, false, imageutils.GetPauseImageName(), podType) </del> <ins> pods := newDensityTestPods(testArg.podsNr, false, imageutils.GetPauseImageName(), podType) </ins> // the controller watches the change of pod status controller := newInformerWatchPod(f, mutex, watchTimes, podType)

Kubelet and System-reserve are implemented on Windows along with pod/container reserves which can help prevent overprovisioning, but we still need to test OOM behavior. We need a test case that will: kubelet & system reserve pods until node is full something that will take up system ram so that OOM is triggered /sig windows /milestone v1.14
I have started to look into this test case and have a few questions: What is the expected behavior we are testing for? What does "run something that will take up system ram" mean? Does that mean run a pod that uses up memory or run a system service that uses memory? After setting the reserves and as mentioned in we should be able to schedule pods until full and then once we trigger OOM (either by pod or system service) kubelet should continue running becuase of the reserves? Do the two settings need to be tested independently or they should be tested together? I found that seems to be doing something similar but it is in node-e2e. Is sig-windows running node-e2e tests currently?
/assign
GitHub didn't allow me to assign the following users: jsturtevant. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After researching this further and talking with I found that the does test the allocatable memory but uses CGroups to set reserves to test the reported vs actual. As windows does not have CGroups this test will not port as is. There is and the is not implemented so will not work either. The proposed testing solution would be two parts: kubelet & system reserve and verify allocatable is reporting properly. Schedule pods with known memory consumption values and verify the allocatable is correclty updated.Schedule until full and verify scheduling is not allowed. kubelet & system reserve and verify allocatable is reporting properly. Schedule Schedule until full and verify scheduling is not into machine and run process that consumes memory and verify kubelet and pods remain running. There is an open question on the second test as to whether kubelet and will be have properly.

If it does not we will submit test 1 and open an issue to resolve 2.
I would suggest adding a third case that would test the behavior when excessive memory pressure is exerted on the node from the pods. This is perhaps a more common use case than 2 since the pod workloads are more likely to exhibit greater resource variability than node system processes. It would also carry over as a test for when/if node allocatable enforcement get implemented for windows nodes. Additionally, to complement this work would it be suitable to test windows pod memory limit enforcement? Or is this tested elsewhere?
Agreed this is a more common case. I am not sure that it is. Is this tested on linux somewhere?
I am seeing strange numbers reported when performing a test case. I scheduled a few pods to test memory limits and then schedule a pod with no memory limit and a process that will consume all of the available memory 500mb at a time: After the pod consumes all available memory, When I compare Node Allocatable/Node usage/Pod Usage I am not seeing the numbers add up properly: My node is staying online and kubelet is reporting no memory pressure. The total node memory consumption is over the Allocatable memory. What is the expected behavior and what should the test look like?
I believe node allocatable will always just be capacity minus reserves (system and kubelet) - So in this scenario it looks like you have and/or set? The memory % on the node is relative to the allocatable amount, and still seems to be under the total memory capacity - are you sure that all available memory has been consumed by your oom pod? The node should be tainted with memory-pressure when a hard or soft eviction threshold is met (). What do you have set for and ?
Thanks for the clarification. I currently have these settings: And on the machine I am seeing most memory being used: If I understand correctly, I start to see memory-pressure once I get into the setting above. In that case I should start to see pods get evicted, so for a valid test I could adjust the values to be easier to hit then test to see that pods get evicted properly which would be similar the .

I am going to adjust a few of the settings and manually test that this is working then see what it will take to port the PriorityMemoryEviction, which currently uses cgroups which won't work on windows. This combined with testing that Memory Allocatable is being properly set should give us a good test coverage for windows on this scenario.
I performed manual testing and found that even when memory was used up on the machine the "Memory Pressure" notification was not fired. After looking deeper I found is not implemented for windows. The , which uses cgroups, provides the . This means even though is being it isn't used. On Windows, Node allocatable when used with limits and requests does prevent the node from over scheduling. I believe the the test and  (this only shows cpu and I could not find a test that demonstrated memory) demonstrate this works. But this doesn't stop a pod from consuming all the memory on the node and since eviction (hard/soft/node allocatable) is not implemented on windows this means a node can end up resource starved. I think the only valid test to be written for this scenario, as it stands now, is validating that Node Allocatable is being reported properly. Beyond that there is no functionality that can be validated around OOM behavior for the windows node.
Yeah, that sounds to be in agreement with what I've observed on the windows nodes that I run. To prevent resource starvation (as a workaround), would setting mem/cpu requests equal to mem/cpu limits for all windows pods work? I think the request / limit values get passed down into the docker daemon when it starts a container, so is it possible that docker could enforce the memory limit?
Yes, setting limits==requests would help the scheduler not over provision the nodes. I think the best way to approach this particular issue for 1.14 is to make sure docs call out what works/doesn't work and give guidance to on how to prevent resource starvation on nodes. Is there a place in the docs that would make sense for this information?

The call outs would be: , , and flags do update Node Allocatable is not implemented and are not implemented Condition is not implemented in Windows There are no OOM actions taken by kubelet As a work around to previous, and should always be equal to prevent memory resource starvation as this allows the scheduler to to use limits and Kubelet running on the windows node does not have memory restrictions. and system-reserve does not set limits on kubelet or processes running the host. This means kubelet or a process on the host could cause memory resource starvation outside the and scheduler
thank you for the thorough investigation


Question: Create a LoadBalancer service with the specified name.`) serviceLoadBalancerExample = templates.Examples(` <del> # Create a new nodeport service named my-lbs </del> <ins> # Create a new LoadBalancer service named my-lbs </ins> kubectl create service loadbalancer my-lbs --tcp=5678:8080`) )

kubectl create service loadbalancer --help Create a LoadBalancer service with the specified name. Examples: # Create a new nodeport service named my-lbs kubectl create service loadbalancer my-lbs --tcp=5678:8080 $ kubectl version Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-16T06:20:04Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}


Question: config:      new(kubeproxyconfig.KubeProxyConfiguration), healthzPort: ports.ProxyHealthzPort, metricsPort: ports.ProxyStatusPort, <del> CleanupIPVS: true, </del> errCh:       make(chan error), } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-proxy don’t clean ipvs rules with . What you expected to happen: kube-proxy clean ipvs rules with by default. How to reproduce it (as minimally and precisely as possible): Run kube-proxy in ipvs mode, and clean rules with . Anything else we need to know?: Based on , should cleanup all rules by default, including ipvs rules. And also retains functionality. Environment: Kubernetes version (use ): v1.20.0-bate.0 Cloud provider or hardware configuration: OS (e.g: ): centos7 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network
it seems that this flag entered deprecation, so I think that recovering functionaly is not the right approach you were involved in the discussion with this flag in , can you take a look at this?
thanks for your reply. The was in , to avoid removing IPVS rules which are not created by kube-proxy. According to , It seems that deleting ipvs rules by default is expected, but it is not so. I think that keeping --cleanup-ipvs is meaningful, unless kube-proxy can only delete the IPVS rules created by itself.
/assign
sorr for the late answer , but the flag must have the same functionality than , actually I can see that it has implemented. In case that the is not working we should fix it, but not recover the deprecated flag
Agree
sorry if I'm missing something. I just try to make the function of as described, not to recover the flag. The problem I see is: The can't flush IPVS. Because the IPVS flush process is still controlled by . Beforce , it was assigned by flag, but now it is not assigned, always false. currently does not have any functionality, just stored by . So, should we keep the function of temporarily by assigning it to ,or just make always flushing IPVS regardless of the ?

I prefer the former because the flag is not completely abandoned :)
it seems that is what the flag description says, you have to use to flush the ipvs entries if that is the case it seems that we are ok, right? no changes needed
In fact, can't flush ipvs entries. The reason is what I described above.
I´m sorry, I think I got it know, the flag is not exposed, but the option is there and initialized to true, this is what I see so, it seems that flush IPVS by default
yes, should flush IPVS by default, but it doesn’t actually
... and finally I got it, so sorry :bow: the ProxyServer is not being instantiated with the flag, despite is set :facepalm: you are more familiar with the IPVS implementation than me, I personally think that, if the flag deprecation has started, we should make that always flush IPVS and totally remove ... it seems aligned with the other proxy implementations
/triage accepted
Actually, after doing some code archaeology, this was the expectation per And the flag clearly mentions that --cleanup must flush IPVS entries too I think that we can go ahead and remove the --cleanup-ipvs flag and make --cleanup to flush ipvs too, right now is a mess and is not working at all , are you going to submit a PR?
thanks for digging up the history on this. That lines up with my recollection and the KEP around kube-proxy's clean up logic
OK, I will create a PR to remove the flag


Question: } return secretName, err } <ins> // check whether mountOptions contain file_mode and dir_mode, if not, append default mode func appendDefaultMountOptions(mountOptions []string) []string { fileModeFlag := false dirModeFlag := false versFlag := false for _, mountOption := range mountOptions { if strings.HasPrefix(mountOption, fileMode) { fileModeFlag = true } if strings.HasPrefix(mountOption, dirMode) { dirModeFlag = true } if strings.HasPrefix(mountOption, vers) { versFlag = true } } allMountOptions := mountOptions if !fileModeFlag { allMountOptions = append(allMountOptions, fmt.Sprintf("%s=%s", fileMode, defaultFileMode)) } if !dirModeFlag { allMountOptions = append(allMountOptions, fmt.Sprintf("%s=%s", dirMode, defaultDirMode)) } if !versFlag { allMountOptions = append(allMountOptions, fmt.Sprintf("%s=%s", vers, defaultVers)) } return allMountOptions } </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): azurefile azurefile 0777 Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:38:40Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Microsoft Azure Container Service OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS Kernel (e.g. ): Linux 4.4.0-28-generic -Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x8664 x8664 x8664 GNU/Linux What happened: I attempted to set up an Azure File Volume using the built-in volume type azureFile. Everything works, except that the files and directories defaults to permission 0777, always. When I tried to manually mount within the container, I could customise dirmode and filemode, meaning that this change should be possible. Reading the source code confirms it - dirmode=0777 and filemode=0777 seem hardcoded. What you expected to happen: Being able to supply dirmode and filemode to the mount. e.g. How to reproduce it (as minimally and precisely as possible): N/A. This is a feature request.

Anything else do we need to know: N/A
This is a big requirement, using an azure file volume for postgresql for instance fails due to permissions as initdb tries to change the permissions to be less restrictive but is unable to.
Evaluating compatibility between a certain setup of a volume mount and an arbitrary piece of software would be the responsibility of the integrator, wouldn't it? It seems a big requirement for K8s itself to have to ensure PostgreSQL can do what it wants. An integrator who tries to use Azure File Storage for PostgreSQL should know better to mount with the correct permission.
The point is that you cannot mount with permissions, they're hard coded to 0777.
Ah. Yeah. Sorry for misunderstanding the comment above. Let's say it is an "important" requirement :)
/sig azure
/assign andyzhangx
I have a for this, use mount options in storage class, below is an example:
Has this change made it into any release yet? I can't seem to find it... Or was the issue resolved in some other way? I'm facing the issue you originally reported, i.e. using a pod volume to an azureFile, and it always mounting with filemode and dirmode of 0755... and I need it to be 0777. Is there a workaround? TIA.
azure file mountOptions feature is available from v1.8.5, here is the example :
Thanks -- I was still having trouble until I the storageClassName: "" to the claim... then realized I was hitting issue which I can avoid by setting storageClassName: "default" to both the pv and pvc, and now everything works. Thanks again!


Question: func GetHostname(hostnameOverride string) string { hostname := []byte(hostnameOverride) if string(hostname) == "" { <del> // Note: We use exec here instead of os.Hostname() because we // want the FQDN, and this is the easiest way to get it. fqdn, err := exec.Command("hostname", "-f").Output() </del> <ins> fqdn, err := exec.Command("uname", "-n").Output() </ins> if err != nil { glog.Fatalf("Couldn't determine hostname: %v", err) }

Actually appending fqdn suffix already causes issue for non-resolvable node names based on today: Today when NodeController "registers" nodes in SyncCloud by getting bare instance names () from GCE, and recreating the fqdn and append it to instance names().


Question: assert.Equal(t, svc.Name, slices[0].Labels[discovery.LabelServiceName]) assert.EqualValues(t, []discovery.EndpointPort{}, slices[0].Ports) assert.EqualValues(t, []discovery.Endpoint{}, slices[0].Endpoints) <ins> expectTrackedResourceVersion(t, r.endpointSliceTracker, &slices[0], "100") </ins> expectMetrics(t, expectedMetrics{desiredSlices: 1, actualSlices: 1, desiredEndpoints: 0, addedPerSync: 0, removedPerSync: 0, numCreated: 1, numUpdated: 0, numDeleted: 0}) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With EndpointSlice feature enabled, after EndpointSliceController creates or updates an EndpointSlice when reconciling a Service, the Service would always be processed one extra time, which is triggered by the EndpointSlice /updated event made by itself during the first processing. What you expected to happen: EndpointSliceController shouldn't reprocess a Service due to the EndpointSlice update made by itself. How to reproduce it (as minimally and precisely as possible): EndpointSlice feature, improve kube-controller-manager log to 5. a nginx deployment a Service backed by the above deployment. kube-controller-manager log: the service will be processed twice in succession. Anything else we need to know?: It's because EndpointSliceController tracks stale resource version after creating/updating an EndpointSlice. Environment: Kubernetes version (use ): v1.18.0-alpha.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: fi export IFS=$'n' <ins> # NOTE: when "go list -e ./..." is run within GOPATH, it turns the k8s.io/kubernetes # as the prefix, however if we run it outside it returns the full path of the file # with a leading underscore. We'll need to support both scenarios for all_packages. </ins> all_packages=( <del> $(go list -e ./... | egrep -v "/(third_party|vendor|staging|generated|clientset_generated)" | sed 's/k8s.io/kubernetes///g') </del> <ins> $(go list -e ./... | egrep -v "/(third_party|vendor|staging|generated|clientset_generated)" | sed -e 's|^k8s.io/kubernetes/||' -e "s|^_${KUBE_ROOT}/?||") </ins> ) linted_packages=( $(cat $linted_file)

Looks like it is chopping off the correct Kube prefix, maybe because of bash 4.0isms.
please let me know if the PR works for you!


Question: labels: k8s-app: kube-dns annotations: <del> seccomp.security.alpha.kubernetes.io/pod: 'docker/default' </del> <ins> seccomp.security.alpha.kubernetes.io/pod: 'runtime/default' </ins> prometheus.io/port: "10054" prometheus.io/scrape: "true" spec:

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Apply security best practices for both kube-dns and core-dns. Low hanging fruit: [x] Run as non-root & disallow privilege escalation [x] Run with the default seccomp profile (i.e. ) [x] ReadOnlyRootFilesystem [x] Avoid unnecessary HostPath volumes [x] (n/a) More advanced: [ ] Reduce dependencies in the base image () [ ] Custom seccomp profile [ ] Custom AppArmor profile [x] Drop unneeded capabilities[x] Run as non-root & disallow privilege escalation [x] Run with the default seccomp profile (i.e. ) [x] ReadOnlyRootFilesystem [x] Avoid unnecessary HostPath volumes [x] (n/a) More advanced: [ ] Reduce dependencies in the base image () [ ] Custom seccomp profile [ ] Custom AppArmor profile [x] Drop unneeded capabilities This is based on the long term issue and implemented by PR . Why is this needed: Our system pods should run using security best practices, both to enhance cluster security and serve as examples of best practices to users.
/sig auth /area security


Question: Name: "example-named-port", }, Spec: v1.ServiceSpec{ <del> Selector: map[string]string{labelShared12: labelValue}, </del> <ins> Selector:                 map[string]string{labelShared12: labelValue}, PublishNotReadyAddresses: true, </ins> Ports: []v1.ServicePort{{ Name:       "http", Port:       80,

Which jobs are flaking: ci-kubernetes-kind-e2e-parallel pr:pull-kubernetes-e2e-gce pr:pull-kubernetes-e2e-gce-ubuntu-containerd pr:pull-kubernetes-e2e-gce-ubuntu-containerd ci-kubernetes-kind-ipv6-e2e-parallel Which test(s) are flaking: Second flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know: /sig network /assign
needs investigation since we're switching kube-proxy to endpointslice in this release
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
made me think of , but that's in already.
It's possible it's a test timeout that's just too close, but we need verification of that checking the controller and apiserver logs to make sure there isn't a latent bug
Thanks for catching this! It looks like the timeouts are actually associated with waiting for 2 pods to be ready, not actually anything related to EndpointSlice functionality - timeout is here: Given that, it seems like the obvious solution would be to update the Services to set to and update this block to wait for the Pods to have IPs, not for them to be ready. If that sounds reasonable, I'll work on getting a fix in tomorrow.
/remove-triage unresolved
If that is still exercises the same conditions (I vaguely remember publishNotReadyAddresses changing code paths in the endpoints controller... does it do the same in the endpointslice controller?), that seems reasonable to me.
I think the code path is very similar here, the only real difference should be how the EndpointSlice controller calculates readiness. I've gone ahead and filed a PR that uses this approach, hopefully it helps:
This flaked again this morning /reopen
Reopened this issue. <detailsIn response to : This flaked again this morning Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This should be /priority critical-urgent given
The top level error message does not make this clear but the issue here is that too many EndpointSlices are being created - 3 instead of 2.
Interesting... that condition should not persist for O(minutes), right?

that seems like it might be a bug, not just a test issue
There's potential that it's not consistently merging 2 ports with 1 endpoint into a single slice and instead creating a slice per port. It also could represent a larger bug. Trying to find a way to consistently reproduce this.
Despite my best efforts I have been unable to reproduce this on my own e2e cluster. I've had 10 test runs going at the same time repeatedly without any luck, any ideas for how to reproduce flaky test failures local would be welcome. In the meantime, I've created a follow up PR that will provide more helpful information if/when this test fails again: My leading theory right now is that this could be a caching issue. The EndpointSlice controller runs a sync, sees that we don't have an EndpointSlice for the given port combination in cache and then goes ahead and creates one, even though a matching EndpointSlice already exists and has not been cached yet. That would result in an annoying situation where an essentially duplicate EndpointSlice was created due to a stale cache. I would think the EndpointSliceTracker would resolve this issue for us though, so not sure. Getting the above PR in will help validate this theory. It's worth noting that if this is actually a duplicate EndpointSlice, clients like kube-proxy will handle this seamlessly because they already have to handle potential duplication in EndpointSlices. That would potentially explain why a test failure like this does not seem to be causing other things to break.
It appears that my theory was correct as witnessed here: I create a .

The key points I've noticed so far: 1 EndpointSlice references pod2 (example-named-port-f7vq5) 2 EndpointSlices reference pod1 (example-named-port-fkgcc and example-named-port-vfp2q) 2 EndpointSlices were created at "2020-07-28T04:57:51Z": example-named-port-f7vq5 (references pod2) example-named-port-vfp2q (references pod1) 1 EndpointSlice was created 1 second later ("2020-07-28T04:57:52Z") example-named-port-fkgcc (references pod1) All 3 EndpointSlices are: On generation 1 - no updates have occurred With a state of ready: true The 2 EndpointSlices referencing pod1 are effectively identical  Of course this is something the EndpointSliceTracker should be handling for us, but even that approach seems less than ideal here. It wouldn't actually prevent the duplicate EndpointSlices from being created, it should just be cleaning up duplicates after they happen. This combined with the race conditions referenced in make me think we may want to find a way to slow down the sync cycle for each Service here. This also feels at least tangentially related to the need for better rate limiting in both this and the Endpoints controller discussed in . /cc
Hmm - actually I don't understand why this is happening. Maybe I'm missing something, but I don't think slowing down controller is needed to solve this problem. Even though the controller usually is not lagging my more than 1s, it is possible that it can be lagging by a minute or more in pathological cases. So we can't just rely on the fact that it's guaranteed to be synced. Can you explain it a bit deeper why this is happening? i.e. some more detailed timeline like: T1: state of cache is XYZ T2: controller computes the ES x1, x2 T3: state of cache in ABC T4: controller computes ES y1 [why?]
Agree, I'm not in favor of slowing down the controller globally to address this. Delaying resync of a particular service until we observe the existence of new endpointslices we know we created (with some max delay) could be reasonable.

That's exactly where I was hoping to end up here, although I'm not quite sure how to do that cleanly yet. To be clear, this is not just to fix this flaky test. This flaky test is a symptom of a much larger and more annoying issue with the current implementation. The EndpointSlice controller is consistently running into errors where it's trying to update EndpointSlices that are stale. This is caused by follow up calls to running before the informer cache is up to date, which happens all the time during rolling updates where rapid churn of EndpointSlices is expected. Although this eventually resolves itself, it's causing a lot more load on API Server than necessary and can potentially result in some bad states where an update fails but a delete succeeds. In this specific instance, it looks like a syncService run is occurring after the first 2 EndpointSlices have been created but before the one referencing pod1 is in the informer cache. That results in the controller creating a new EndpointSlice to reference pod1 and thus we end up with a duplicate. If anything in the future triggers another syncService call that would be resolved. I'd initially thought that EndpointSliceTracker would be thing to trigger that follow up syncService, but I was wrong. That tracker just tracks the latest resource version for each EndpointSlice the controller has created/updated and compares that with EndpointSlices received through create/update events. If the controller doesn't think that EndpointSlice should exist or the controller's resource version for that slice does not match it triggers another syncService. In this case neither of those conditions would be met so no additional syncService call would be triggered. So that's a long way of saying that I think our best possible solution here is to introduce some delay between when Services can be synced. I think the easiest solution that would solve 99% of our problems would be to add something like a 1s delay in between syncs simply by delaying when was called. This would not be ideal, but it would result in an immediate improvement and be easy to backport. It's worth noting that although duplicate EndpointSlices like this or even attempts to update a stale EndpointSlice are not ideal, they are not actually invalid and will eventually resolve themselves. Consumers of this API already have to do some form of deduping.

With that said, I'm hopeful that there's a clean way to implement a solution that actually waited for each EndpointSlice change the controller has made to be reflected in the cache because that would be a better solution. I'll spend some time looking into possible implementations here.
I've created a draft PR that contains my initial progress towards the approach I described above. I don't particularly love this approach so I'm very open to other ways we could solve this issue. It's also still very much a WIP and has significant work remaining. I opened the PR early just so I could get some initial feedback on the direction and potentially inspire other ideas around approaches we could take here.
is still the best approach I can come up with here. It is ready for review if anyone has time to take a look.
Quick update here, I've limited the scope here to the following changes as discussed in Update the flaky test to consider duplicate EndpointSlices valid. Update the docs to make it very clear that consumers of the EndpointSlice API will have to do deduplication (as kube-proxy already does). Update the controller error handling to return immediately if a create, update, or delete errors (in that order). Reviews would be appreciated on all of the above.
Reopening since is not yet merged.
Thanks for catching that Closing now that is merged. /close
Closing this issue. <detailsIn response to : Thanks for catching that Closing now that is merged. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "github.com/onsi/ginkgo" "google.golang.org/grpc/codes" <del> appsv1 "k8s.io/api/apps/v1" </del> v1 "k8s.io/api/core/v1" rbacv1 "k8s.io/api/rbac/v1" storagev1 "k8s.io/api/storage/v1"

newer hostpath deployments but removed the csi-external-health-monitor-agent because it seemed to cause overhead without being needed for the tests. We should re-enable it and check whether that really causes tests to fail because of the extra load.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /area test


Question: expectNeedsCleanup bool }{ { <del> desc:               "service without finalizer without timestamp", </del> <ins> desc:               "service without finalizer", </ins> svc:                &v1.Service{}, expectNeedsCleanup: false, }, { <del> desc: "service without finalizer with timestamp", </del> <ins> desc: "service with finalizer without timestamp without LB", </ins> svc: &v1.Service{ ObjectMeta: metav1.ObjectMeta{ <del> DeletionTimestamp: &metav1.Time{ Time: time.Now(), }, </del> <ins> Finalizers: []string{servicehelper.LoadBalancerCleanupFinalizer}, }, Spec: v1.ServiceSpec{ Type: v1.ServiceTypeNodePort, </ins> }, }, <del> expectNeedsCleanup: false, </del> <ins> expectNeedsCleanup: true, </ins> }, { <del> desc: "service with finalizer without timestamp", </del> <ins> desc: "service with finalizer without timestamp with LB", </ins> svc: &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Finalizers: []string{servicehelper.LoadBalancerCleanupFinalizer}, }, <ins> Spec: v1.ServiceSpec{ Type: v1.ServiceTypeLoadBalancer, }, </ins> }, expectNeedsCleanup: false, },

What happened: Service Load balancer may not be cleaned up on type change if below sequence happen: type changes from to controller restarts/crashes before it gets a chance to process the update from 1. controller starts up again and sees the service doesn't want load balancer, hence skips it. load balancer resource sticks around even if it is no longer needed by the service. Note that this is a different issue than , as load balancer resource will still be cleaned up when the service object is deleted, just not when the type change happens. Anything else we need to know?: This is a corner case that is not covered by the initial service load balancer finalizer implementation (). This can probably be fixed in a trivial way --- make sure the handler takes in service with non-LoadBalancer type but has finalizer attached: I will put more thoughts later. cc
/sig network
Can i reserve this issue in advance? :) I'd like to take this issue as my first issue.
Sure thing, feel free to work on this and glad that you can help :)
Do you mean attach finalizer for service that non-LoadBalancer type in AddFunc? I'm new to kubernetes, hope you can guide me. :)
No worries. I actually meant should include service with non-LoadBalancer type && with finalizer attached. As we only attach finalizer to a service only when we are creating a load balancer for it, a finalizer presents on the service (no matter what type it has) indicates there is a load balancer associated. Hence it makes sense that we should clean up the load balancer if the service no longer wants one, even if the service has not been deleted.
To clarify further, "attach finalizer for service that non-LoadBalancer type in AddFunc" would not be what we want --- we shouldn't attach finalizer to service that we will not be creating load balancer for.
After go through service controller part code, i guess you mean add a condition for as follows: . Am i right? I'm not sure when service type changes if can be involved. Can you clarify this ?
if ok && (wantsLoadBalancer(svc) (svc.Spec.Type != v1.ServiceTypeLoadBalancer && servicehelper.HasLBFinalizer(svc))) . Yea this was what I meant.

On service type change, will be involved instead of . That part is already handled. This issue is explicit for the case when controller fails to handle the type change update before it exiting (e.g. crashing, restart, etc.). Putting such detection to is more off a remedy, it means the next time service controller starts up, it can re-syncs all existing services and find out it still need to cleanup load balancer for this service, even if it doesn't have load balancer type anymore.
Thanks . I have sent a PR assined to you. I think it should be checked in function . It is the case that need to clean up whatever the reason is.


Question: // in extensions. These addons are: // - MaxUnavailable during rolling update set to 25% (1 in extensions) // - MaxSurge value during rolling update set to 25% (1 in extensions) <del> // - RevisionHistoryLimit set to 2 (not set in extensions) </del> <ins> // - RevisionHistoryLimit set to 10 (not set in extensions) </ins> // - ProgressDeadlineSeconds set to 600s (not set in extensions) func SetDefaults_Deployment(obj *appsv1beta2.Deployment) { // Default labels and selector to labels from pod template spec.

All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps


Question: "context" "fmt" "os" <del> "os/exec" "path" </del> "regexp" "strings" "time" <ins> "github.com/onsi/ginkgo" "github.com/onsi/gomega" appsv1 "k8s.io/api/apps/v1" </ins> v1 "k8s.io/api/core/v1" rbacv1 "k8s.io/api/rbac/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" <ins> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" </ins> e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" imageutils "k8s.io/kubernetes/test/utils/image" <del> "github.com/onsi/ginkgo" "github.com/onsi/gomega" </del> ) const (

Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing


Question: // Names should come from the REST response, NOT the files expectRC := "replicationcontroller/" + nameRC + "n" expectSVC := "service/" + nameSVC + "n" <del> expect := expectRC + expectSVC if buf.String() != expect { t.Fatalf("unexpected output: %snexpected: %s", buf.String(), expect) </del> <ins> // Test both possible orders since output is non-deterministic. expectOne := expectRC + expectSVC expectTwo := expectSVC + expectRC if buf.String() != expectOne && buf.String() != expectTwo { t.Fatalf("unexpected output: %snexpected: %s OR %s", buf.String(), expectOne, expectTwo) </ins> } }

kubectl create, delete, etc. can already take a List object. kubectl apply can take the list object when it does creation, but it gives an (ugly) error when it's configuring the objects. Then... expected: Actual:
what's you ?
0-alpha.5
Sounds like a bug. Are you interested in taking a look
sure I will take a look during this weekend.
I think the root cause is , the actually is , in the origin , is not , but after the , this object is nil. Would you like to double check this?
I'll take a look.
if you have make sure about this, please let me know. Thanks :)
Is there any ways we could get this into 1.2? It's actually making our config generation quite painful to have to make a separate file for every object. It looks like got pretty far? would you like to make a PR that we can review?
I don't see how. It could have made 1.2 if a PR had been sent within a few days of the issue being filed.
I unassigned all issues assigned to you. Please only assign issues that are actively being worked on, so that others can know which issues are available to be picked up.
My apologies. If I'm not mistaken, this issue was filed after the code freeze for 1.2. I didn't realize that it was time critical or that it could make the release at that time. However, I should have asked. Sorry for the inconvenience. Thank you. I still plan to address those issues for the next release, as discussed, but will assign each one to myself as I work on it.
I created which fixes this. I'm really hoping we can get this into 1.2 since it's a pretty small, isolated fix but a pretty big bug that is complicating our config generation system. Let me know what else I can do to help.
Thanks for the PR. We're short on reviewer bandwidth at the moment. I assigned it to Clayton for review or delegation.


Question: return testexec.InitFakeCmd(&c, cmd, args...) } } <ins> func TestNotMountedBehaviorOfUnmount(t *testing.T) { target, err := ioutil.TempDir("", "kubelet-umount") if err != nil { t.Errorf("Cannot create temp dir: %v", err) } defer os.RemoveAll(target) m := Mounter{withSafeNotMountedBehavior: true} if err = m.Unmount(target); err != nil { t.Errorf(`Expect complete Unmount(), but it dose not: %v`, err) } if err = tryUnmount(target, m.withSafeNotMountedBehavior, time.Minute); err != nil { t.Errorf(`Expect complete tryUnmount(), but it does not: %v`, err) } // forceUmount exec "umount -f", so skip this case if user is not root. if os.Getuid() == 0 { if err = forceUmount(target, m.withSafeNotMountedBehavior); err != nil { t.Errorf(`Expect complete forceUnmount(), but it does not: %v`, err) } } } </ins>

A status of Pod mounted NFS PV remain as terminating when a node shutdown and restart. A Pod is completely terminated. Deploy a Pod mounted NFS PV Shutdown the node where the Pod is running Wait for 5 min and the Pod status become Restart the node The Pod status is still and the Pod is never terminated completely <detailsPrepare a NFS Server.(I used ) Deploy a Pod mounted NFS PV like this. Shutdown the node where the Pod is running.(In this case ) And the status of node02 becomes . After 5 min later(by default), the Pod status becomes . (The time a Pod becomes status depends on .) Then, restart the node and the status of node02 becomes . But the Pod status is still and the Pod is never terminated completely. It seems that a faild to delete Pod because of trying to unmount a not mounted directory. Force delete Pod. Delete Pod's directory on the node. </detailsNo response <details</details<detailsnone(on premise) </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig node
It dose not happen using , so it seems to depend on NFS volume plugin.
/assign
I try to build and run kubelet with comment out this logic. Then it seems to get well. related: This change checks the result of instead of the mount point check. However, check logic is not implemented in and (these functions are used in NFS volume plugin), despite it is done in .
/remove-sig node /sig storage
Reopening, fix had to be reverted due to unit tests requiring root permissions ()
I am terribly sorry... Please wait a moment. I'll try to fix it.
no worries, just get to review the rework, making sure the units tests don't require root permissions
Thank you! Fixed in .
Are there any plans to backport this to older Kubernetes versions? This seems like a critical fix.


Question: pods := getSummaryTestPods(f, numRestarts, pod0, pod1) f.PodClient().CreateBatch(pods) <ins> ginkgo.By("restarting the containers to ensure container metrics are still being gathered after a container is restarted") </ins> gomega.Eventually(func() error { for _, pod := range pods { err := verifyPodRestartCount(f, pod.Name, len(pod.Spec.Containers), numRestarts)

Which jobs are failing: ci-cri-containerd-node-e2e Which test(s) are failing: Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api Since when has it been failing: 2020 10-12 09:04 PDT Since Testgrid link: Reason for failure: Capability set incorrectly. CAPNETRAW should be NETRAW. I1013 18:39:19.058] Oct 13 18:10:51.403: INFO: At 2020-10-13 18:09:34 +0000 UTC - event for stats-busybox-0: {kubelet tmp-node-e2e--cos-81--1210-0} Failed: Error: failed to create containerd task: OCI runtime create failed: starting container process caused: unknown capability "CAPCAPNET_RAW": unknown Anything else we need to know: I have no idea where this is documented. It's a best guess.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
it's a node_e2e test so /sig node
Seems like there's some dispute on who is doing the correct thing.
I would like to have a try.
please see discussion in


Question: func TestPartialRetryableFailure(t *testing.T) { fake := &fakeDiscovery{ groupList: &metav1.APIGroupList{ <del> Groups: []metav1.APIGroup{{ Name: "astronomy", Versions: []metav1.GroupVersionForDiscovery{{ GroupVersion: "astronomy/v8beta1", Version:      "v8beta1", }, { GroupVersion: "astronomy2/v8beta1", Version:      "v8beta1", }}, }}, </del> <ins> Groups: []metav1.APIGroup{ { Name: "astronomy", Versions: []metav1.GroupVersionForDiscovery{{ GroupVersion: "astronomy/v8beta1", Version:      "v8beta1", }}, }, { Name: "astronomy2", Versions: []metav1.GroupVersionForDiscovery{{ GroupVersion: "astronomy2/v8beta1", Version:      "v8beta1", }}, }, }, </ins> }, resourceMap: map[string]*resourceMapEntry{ "astronomy/v8beta1": {

Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: [sig-api-machinery] Garbage collector should not be blocked by dependency circle [Conformance] [sig-api-machinery] Garbage collector should orphan RS created by deployment when deleteOptions.PropagationPolicy is Orphan [Conformance] [sig-api-machinery] Garbage collector should support cascading deletion of custom resources [sig-api-machinery] Garbage collector should keep the rc around until all its pods are deleted if the deleteOptions says so [Conformance] [sig-api-machinery] Garbage collector should delete RS created by deployment when not orphaning [Conformance] [sig-api-machinery] Garbage collector should delete pods created by rc when not orphaning [Conformance] [sig-api-machinery] Garbage collector should delete jobs and pods created by cronjob Testgrid link: Reason for failure: timeout waiting for deletion to complete Anything else we need to know: seems to have gotten much worse on/after 1/24/2020: See the time period on 2/11 for lots of examples number 2 flake according to /sig api-machinery
from the failure in gc events related to the test: test timed out: gc events related to the test:
cc
/assign thank you for fixing it!
top flake according to
looking at the debug logs from the last several failures, filtering remaining pods to name+finalizers+ownerReferences with: jq '[.[] | {"metadata":{"name","finalizers":(.metadata.finalizers // []), "ownerReferences": .metadata.ownerReferences}}]'
no consistent pattern between failures, GC just seems to have been running slower and sometimes timed out the test. tracked likely performance degradation to , fix in
note that should return performance to pre-1/24 levels, but this test was flaky even before that - *circle
Further debugging of GC lag issues in kube-controller-manager in e2e (after merged) Combined log of e2e+kcm+apiserver for the timeout test failure from is at The RC is deleted by the e2e test: GC starts running attemptToDelete on the rc and all pods within a second: attemptToDeleteItem on the RC (started at 17:37:28.)

takes ~17 seconds to process (end result is calling processDeletingDependentsItem on all the child pods): attemptToDelete on the pods (started at 17:37:28. - 17:37:28.) also takes ~17 seconds to complete (end result is GC making a DELETE API call on each pod): 2 seconds of ~17 second delay is the GC controller rebuilding its restmapper via discovery: The other 15 seconds are unaccounted for. Then, GC reacts to changing API group availability by performing its periodic resync of informers, which fails once (because another parallel e2e test removed a CRD at the same time), retries, takes a total of 39 seconds, and blocks item processing until complete: After resyncing monitors, it immediately adds all the items that received updates or were deleted while it was resyncing, including the 10 deleted pods (which no longer exist) and their parent RC: This time there is a ~16 seconds delay from attemptToDelete starting to handle the RC (at 17:38:26.) and the foreground deletion finalizer being removed: But the e2e test has already failed:
cc
The informer resync is not unexpected, because we have lots of parallel e2e tests that add/remove CRDs. The extra time for GC to recover from an API group sync failure (39 seconds, in this case) is not included in most of the GC e2e timeout periods. The 17 and 15 second execution times for attemptToDelete are worthy of more investigation. Combined, they made GC easily exceed the 60 second period (actual time: the test waits for the RC to be removed
adding fine-grained trace.LogIfLong statements to attemptToDelete in particular might be helpful in revealing exactly where we are spending so much time
Tried this out with The long delays outside of sync.Monitors are almost entirely due to the latency of I'm guessing that is fetching from discovery in this codepath for the same reason does: another parallel e2e test removed a CRD at the same time. more details:
Traced this down to the discovery 's cache refresh. See for a fix.


Question: processPod := func(existingPod *v1.Pod) error { existingPodNode, err := ipa.info.GetNodeInfo(existingPod.Spec.NodeName) if err != nil { <del> if apierrors.IsNotFound(err) { klog.Errorf("Node not found, %v", existingPod.Spec.NodeName) return nil } return err </del> <ins> klog.Errorf("Node not found, %v", existingPod.Spec.NodeName) return nil </ins> } existingPodAffinity := existingPod.Spec.Affinity existingHasAffinityConstraints := existingPodAffinity != nil && existingPodAffinity.PodAffinity != nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: This switches NodeLister to scheduler cache for . However, the new implementation in the cache never returns IsNotFound error any more. Rather, it returns a normal error: However, we rely on the error type for logging, for example: What you expected to happen: The cache should return IsNotFound error if applicable. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig scheduling cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi. I'd like to work on this for #Hacktoberfest, and for my very first open-source contribution.
Hello, I would love to contribute too in case. I'm already looking at the code in fact :D
Thanks all for the interest, will assign to since he expressed interest first. /assign
I was browsing around the rest of the implementation of the cache and noticed that none of the other accessors return . Also, I couldn't determine if ever returned anything other than an . I'm new here, so I may be doing it wrong. Are you sure the behavior changed, and are you sure returning is expected? Should the other accessors be changed, too?
Let's look at previous release. The object is initialized in [] where it creates a .

Now let's take a look at implementation: There you can see that it calls NodeLister Get() method, which can return a IsNotFound error.
Hello, is this issue being solved ? If not, i am interested in solving this issue.
Thanks but I am working on this currently.
, if you haven't started your work yet, let me know if I can raise a PR for this issue. I have the change ready.
I started, but haven’t been able to finish creating the PR. Go ahead and submit yours. Thanks. Sent from my iPhone


Question: "kubelet_resources_test.go", "kubelet_test.go", "kubelet_volumes_test.go", <ins> "networks_test.go", </ins> "oom_watcher_test.go", "pod_container_deletor_test.go", "pod_workers_test.go",

This is a sub issue for
cc can you please assign this to me? Thanks.
, please create PR directly and cc to Tim for review.
/sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: match: __default_go_runner_version= - name: "k8s.gcr.io/pause" <del> version: 3.4 </del> <ins> version: 3.5 </ins> refPaths: - path: build/pause/Makefile match: TAG =

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Run the linux pause container as non-root user Why is this needed: The image is being flagged by scanning tools as not setting a nonroot user in the image definition. While the image is based on scratch which provides some coverage from exploiting vulnerabilities in the image that may escalate privileges it is still a good practice to explicitly set the user in the Dockerfile. There are two options - 1 - Use a base image that can add a user and then copy /etc/passwd and set the user 2 - Switch the image to a base image like distroless/static:nonroot
/assign /sig release /area release-eng dependency security /milestone v1.20 ref: , ,
/sig security
What do you think about this? Please also see the discussion in slack which seems to imply that we are just doing this to "making scanners happy"
cc
Just FYI: There is no need to create a user just to use it with , use a uid is good enough. Example run: Still, works too.
That's good to know. Not sure why I've never tried it or if I had it might not have worked then. Have some places I can clean this up
Agreed that this doesn’t seem like a meaningful security improvement, but it does seem like an opportunity to set a good example of not using root when it isn’t needed. Adding ‘USER :’ to the existing dockerfile (or some similar fix; I picked due to long UNIX tradition) “should” be harmless... but in the past pause image changes have been fraught. do you feel like y’all have good arrangements in place to test a modified pause container rigorously enough to feel good about it?
/cc
Heyho I wanted to check how it is going? The code freeze is starting 12.11.2020 (about 3-4 weeks from now) and while there is still plenty of time, we want to ensure that each relevant issue has a chance to be considered. As the issue is tagged for 1.20, is it still planned for this release?
I thought I had done everything that needed to be completed for this. Is there anything left? Just let me know.
Do we still need this issue or we can close it?
This change should be made from a security best practices stand point.

I'll leave it up to you guys if you think you want to do it in the future or not.
Can you check this? Shall we move this to 1.21?
Let's keep this open and kick the can to the next milestone. It's still important, but we're out of time for v1.20. /milestone v1.21
I'm not sure if that is too trivial, but I gave the proposal a test-shot in
/unassign /assign
I am not a container security expert. It still seems unclear to me that this is actually a container best practice in this case, this image is not supported to run as a user image, it's an implementation detail of pods, with ~zero user control over how it runs, AIUI. I'm also not sure if this change is a problem for PID reaping? I'd like / someone from SIG security to review these changes, I can weigh in more usefully WRT the build / update process however
I can't come up with a threat model under which making this change prevents an attack, mostly because the behavior of the pause container is so trivial. It seems to be a nice-to-have: it should be harmless, and it helps us set a better example of running containers as non-root when root is not really needed. As far as I understand PID reaping, running pause as non-root shouldn't be a problem even with shareProcessNamespace, because if it were I would expect similar problems in normal Linux with reaping suid children.
Hello! This Issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting March 9th, 2021 (about 1 week from now). As the Issue is tagged for 1.21, is it still planned for this release?
any update on this issue or for which milestone it will be targeted?
is not within the v1.21 milestone. I have to continue some tests to ensure that the change is the right move forward.
/milestone clear
I found this thread because I have a Kubernetes controller running some containers as UID . This turns up in an audit for processes running with sketchy UIDs. Please don't ship software that runs processes on UIDs that are not defined users.


Question: <del> /* Copyright 2015 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // Package chaosclient makes it easy to simulate network latency, misbehaving // servers, and random errors from servers. It is intended to stress test components // under failure conditions and expose weaknesses in the error handling logic // of the codebase. package chaosclient import ( "errors" "fmt" "log" "math/rand" "net/http" "reflect" "runtime" "k8s.io/apimachinery/pkg/util/net" ) // chaosrt provides the ability to perform simulations of HTTP client failures // under the Golang http.Transport interface. type chaosrt struct { rt     http.RoundTripper notify ChaosNotifier c      []Chaos } // Chaos intercepts requests to a remote HTTP endpoint and can inject arbitrary // failures. type Chaos interface { // Intercept should return true if the normal flow should be skipped, and the // return response and error used instead. Modifications to the request will // be ignored, but may be used to make decisions about types of failures. Intercept(req *http.Request) (bool, *http.Response, error) } // ChaosNotifier notifies another component that the ChaosRoundTripper has simulated // a failure. type ChaosNotifier interface { // OnChaos is invoked when a chaotic outcome was triggered. fn is the // source of Chaos and req was the outgoing request OnChaos(req *http.Request, c Chaos) } // ChaosFunc takes an http.Request and decides whether to alter the response. It // returns true if it wishes to mutate the response, with a http.Response or // error. type ChaosFunc func(req *http.Request) (bool, *http.Response, error) // Intercept calls the nested method `Intercept` func (fn ChaosFunc) Intercept(req *http.Request) (bool, *http.Response, error) { return fn.Intercept(req) } func (fn ChaosFunc) String() string { return runtime.FuncForPC(reflect.ValueOf(fn).Pointer()).Name() } // NewChaosRoundTripper creates an http.RoundTripper that will intercept requests // based on the provided Chaos functions. The notifier is invoked when a Chaos // Intercept is fired. func NewChaosRoundTripper(rt http.RoundTripper, notify ChaosNotifier, c ...Chaos) http.RoundTripper { return &chaosrt{rt, notify, c} } // RoundTrip gives each ChaosFunc an opportunity to intercept the request. The first // interceptor wins. func (rt *chaosrt) RoundTrip(req *http.Request) (*http.Response, error) { for _, c := range rt.c { if intercept, resp, err := c.Intercept(req); intercept { rt.notify.OnChaos(req, c) return resp, err } } return rt.rt.RoundTrip(req) } var _ = net.RoundTripperWrapper(&chaosrt{}) func (rt *chaosrt) WrappedRoundTripper() http.RoundTripper { return rt.rt } // Seed represents a consistent stream of chaos. type Seed struct { *rand.Rand } // NewSeed creates an object that assists in generating random chaotic events // based on a deterministic seed. func NewSeed(seed int64) Seed { return Seed{rand.New(rand.NewSource(seed))} } type pIntercept struct { Chaos s Seed p float64 } // P returns a ChaosFunc that fires with a probability of p (p between 0.0 // and 1.0 with 0.0 meaning never and 1.0 meaning always). func (s Seed) P(p float64, c Chaos) Chaos { return pIntercept{c, s, p} } // Intercept intercepts requests with the provided probability p. func (c pIntercept) Intercept(req *http.Request) (bool, *http.Response, error) { if c.s.Float64() < c.p { return c.Chaos.Intercept(req) } return false, nil, nil } func (c pIntercept) String() string { return fmt.Sprintf("P{%f %s}", c.p, c.Chaos) } // ErrSimulatedConnectionResetByPeer emulates the golang net error when a connection // is reset by a peer. // TODO: make this more accurate // TODO: add other error types // TODO: add a helper for returning multiple errors randomly. var ErrSimulatedConnectionResetByPeer = Error{errors.New("connection reset by peer")} // Error returns the nested error when C() is invoked. type Error struct { error } // Intercept returns the nested error func (e Error) Intercept(_ *http.Request) (bool, *http.Response, error) { return true, nil, e.error } // LogChaos is the default ChaosNotifier and writes a message to the Golang log. var LogChaos = ChaosNotifier(logChaos{}) type logChaos struct{} func (logChaos) OnChaos(req *http.Request, c Chaos) { log.Printf("Triggered chaotic behavior for %s %s: %v", req.Method, req.URL.String(), c) } </del>

/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?


Question: <ins> /* Copyright 2014 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package common import ( "fmt" "time" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/test/e2e/framework" imageutils "k8s.io/kubernetes/test/utils/image" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) var _ = Describe("[sig-storage] Projected downwardAPI", func() { f := framework.NewDefaultFramework("projected") // How long to wait for a log pod to be displayed const podLogTimeout = 2 * time.Minute var podClient *framework.PodClient BeforeEach(func() { podClient = f.PodClient() }) /* Release : v1.9 Testname: Projected Volume, DownwardAPI, pod name Description: A Pod is created with a projected volume source for downwardAPI with pod name, cpu and memory limits and cpu and memory requests. Pod MUST be able to read the pod name from the mounted DownwardAPIVolumeFiles. */ framework.ConformanceIt("should provide podname only [NodeConformance]", func() { podName := "downwardapi-volume-" + string(uuid.NewUUID()) pod := downwardAPIVolumePodForSimpleTest(podName, "/etc/podinfo/podname") f.TestContainerOutput("downward API volume plugin", pod, 0, []string{ fmt.Sprintf("%sn", podName), }) }) /* Release : v1.9 Testname: Projected Volume, DownwardAPI, volume mode 0400 Description: A Pod is created with a projected volume source for downwardAPI with pod name, cpu and memory limits and cpu and memory requests. The default mode for the volume mount is set to 0400. Pod MUST be able to read the pod name from the mounted DownwardAPIVolumeFiles and the volume mode must be -r—-—————. */ framework.ConformanceIt("should set DefaultMode on files [NodeConformance]", func() { podName := "downwardapi-volume-" + string(uuid.NewUUID()) defaultMode := int32(0400) pod := projectedDownwardAPIVolumePodForModeTest(podName, "/etc/podinfo/podname", nil, &defaultMode) f.TestContainerOutput("downward API volume plugin", pod, 0, []string{ "mode of file "/etc/podinfo/podname": -r--------", }) }) /* Release : v1.9 Testname: Projected Volume, DownwardAPI, volume mode 0400 Description: A Pod is created with a projected volume source for downwardAPI with pod name, cpu and memory limits and cpu and memory requests. The default mode for the volume mount is set to 0400. Pod MUST be able to read the pod name from the mounted DownwardAPIVolumeFiles and the volume mode must be -r—-—————. */ framework.ConformanceIt("should set mode on item file [NodeConformance]", func() { podName := "downwardapi-volume-" + string(uuid.NewUUID()) mode := int32(0400) pod := projectedDownwardAPIVolumePodForModeTest(podName, "/etc/podinfo/podname", &mode, nil) f.TestContainerOutput("downward API volume plugin", pod, 0, []string{ "mode of file "/etc/podinfo/podname": -r--------", }) }) It("should provide podname as non-root with fsgroup [NodeFeature:FSGroup]", func() { podName := "metadata-volume-" + string(uuid.NewUUID()) uid := int64(1001) gid := int64(1234) pod := downwardAPIVolumePodForSimpleTest(podName, "/etc/podinfo/podname") pod.Spec.SecurityContext = &v1.PodSecurityContext{ RunAsUser: &uid, FSGroup:   &gid, } f.TestContainerOutput("downward API volume plugin", pod, 0, []string{ fmt.Sprintf("%sn", podName), }) }) It("should provide podname as non-root with fsgroup and defaultMode [NodeFeature:FSGroup]", func() { podName := "metadata-volume-" + string(uuid.NewUUID()) uid := int64(1001) gid := int64(1234) mode := int32(0440) /* setting fsGroup sets mode to at least 440 */ pod := projectedDownwardAPIVolumePodForModeTest(podName, "/etc/podinfo/podname", &mode, nil) pod.Spec.SecurityContext = &v1.PodSecurityContext{ RunAsUser: &uid, FSGroup:   &gid, } f.TestContainerOutput("downward API volume plugin", pod, 0, []string{ "mode of file "/etc/podinfo/podname": -r--r-----", }) }) /* Release : v1.9 Testname: Projected Volume, DownwardAPI, update labels Description: A Pod is created with a projected volume source for downwardAPI with pod name, cpu and memory limits and cpu and memory requests and label items. Pod MUST be able to read the labels from the mounted DownwardAPIVolumeFiles. Labels are then updated. Pod MUST be able to read the updated values for the Labels. */ framework.ConformanceIt("should update labels on modification [NodeConformance]", func() { labels := map[string]string{} labels["key1"] = "value1" labels["key2"] = "value2" podName := "labelsupdate" + string(uuid.NewUUID()) pod := projectedDownwardAPIVolumePodForUpdateTest(podName, labels, map[string]string{}, "/etc/podinfo/labels") containerName := "client-container" By("Creating the pod") podClient.CreateSync(pod) Eventually(func() (string, error) { return framework.GetPodLogs(f.ClientSet, f.Namespace.Name, podName, containerName) }, podLogTimeout, framework.Poll).Should(ContainSubstring("key1="value1"n")) //modify labels podClient.Update(podName, func(pod *v1.Pod) { pod.Labels["key3"] = "value3" }) Eventually(func() (string, error) { return framework.GetPodLogs(f.ClientSet, f.Namespace.Name, pod.Name, containerName) }, podLogTimeout, framework.Poll).Should(ContainSubstring("key3="value3"n")) }) /* Release : v1.9 Testname: Projected Volume, DownwardAPI, update annotation Description: A Pod is created with a projected volume source for downwardAPI with pod name, cpu and memory limits and cpu and memory requests and annotation items. Pod MUST be able to read the annotations from the mounted DownwardAPIVolumeFiles. Annotations are then updated. Pod MUST be able to read the updated values for the Annotations. */ framework.ConformanceIt("should update annotations on modification [NodeConformance]", func() { annotations := map[string]string{} annotations["builder"] = "bar" podName := "annotationupdate" + string(uuid.NewUUID()) pod := projectedDownwardAPIVolumePodForUpdateTest(podName, map[string]string{}, annotations, "/etc/podinfo/annotations") containerName := "client-container" By("Creating the pod") podClient.CreateSync(pod) pod, err := podClient.Get(pod.Name, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred(), "Failed to get pod %q", pod.Name) Eventually(func() (string, error) { return framework.GetPodLogs(f.ClientSet, f.Namespace.Name, pod.Name, containerName) }, podLogTimeout, framework.Poll).Should(ContainSubstring("builder="bar"n")) //modify annotations podClient.Update(podName, func(pod *v1.Pod) { pod.Annotations["builder"] = "foo" }) Eventually(func() (string, error) { return framework.GetPodLogs(f.ClientSet, f.Namespace.Name, pod.Name, containerName) }, podLogTimeout, framework.Poll).Should(ContainSubstring("builder="foo"n")) }) /* Release : v1.9 Testname: Projected Volume, DownwardAPI, CPU limits Description: A Pod is created with a projected volume source for downwardAPI with pod name, cpu and memory limits and cpu and memory requests. Pod MUST be able to read the cpu limits from the mounted DownwardAPIVolumeFiles. */ framework.ConformanceIt("should provide container's cpu limit [NodeConformance]", func() { podName := "downwardapi-volume-" + string(uuid.NewUUID()) pod := downwardAPIVolumeForContainerResources(podName, "/etc/podinfo/cpu_limit") f.TestContainerOutput("downward API volume plugin", pod, 0, []string{ fmt.Sprintf("2n"), }) }) /* Release : v1.9 Testname: Projected Volume, DownwardAPI, memory limits Description: A Pod is created with a projected volume source for downwardAPI with pod name, cpu and memory limits and cpu and memory requests. Pod MUST be able to read the memory limits from the mounted DownwardAPIVolumeFiles. */ framework.ConformanceIt("should provide container's memory limit [NodeConformance]", func() { podName := "downwardapi-volume-" + string(uuid.NewUUID()) pod := downwardAPIVolumeForContainerResources(podName, "/etc/podinfo/memory_limit") f.TestContainerOutput("downward API volume plugin", pod, 0, []string{ fmt.Sprintf("67108864n"), }) }) /* Release : v1.9 Testname: Projected Volume, DownwardAPI, CPU request Description: A Pod is created with a projected volume source for downwardAPI with pod name, cpu and memory limits and cpu and memory requests. Pod MUST be able to read the cpu request from the mounted DownwardAPIVolumeFiles. */ framework.ConformanceIt("should provide container's cpu request [NodeConformance]", func() { podName := "downwardapi-volume-" + string(uuid.NewUUID()) pod := downwardAPIVolumeForContainerResources(podName, "/etc/podinfo/cpu_request") f.TestContainerOutput("downward API volume plugin", pod, 0, []string{ fmt.Sprintf("1n"), }) }) /* Release : v1.9 Testname: Projected Volume, DownwardAPI, memory request Description: A Pod is created with a projected volume source for downwardAPI with pod name, cpu and memory limits and cpu and memory requests. Pod MUST be able to read the memory request from the mounted DownwardAPIVolumeFiles. */ framework.ConformanceIt("should provide container's memory request [NodeConformance]", func() { podName := "downwardapi-volume-" + string(uuid.NewUUID()) pod := downwardAPIVolumeForContainerResources(podName, "/etc/podinfo/memory_request") f.TestContainerOutput("downward API volume plugin", pod, 0, []string{ fmt.Sprintf("33554432n"), }) }) /* Release : v1.9 Testname: Projected Volume, DownwardAPI, CPU limit, node allocatable Description: A Pod is created with a projected volume source for downwardAPI with pod name, cpu and memory limits and cpu and memory requests.  The CPU and memory resources for requests and limits are NOT specified for the container. Pod MUST be able to read the default cpu limits from the mounted DownwardAPIVolumeFiles. */ framework.ConformanceIt("should provide node allocatable (cpu) as default cpu limit if the limit is not set [NodeConformance]", func() { podName := "downwardapi-volume-" + string(uuid.NewUUID()) pod := downwardAPIVolumeForDefaultContainerResources(podName, "/etc/podinfo/cpu_limit") f.TestContainerOutputRegexp("downward API volume plugin", pod, 0, []string{"[1-9]"}) }) /* Release : v1.9 Testname: Projected Volume, DownwardAPI, memory limit, node allocatable Description: A Pod is created with a projected volume source for downwardAPI with pod name, cpu and memory limits and cpu and memory requests.  The CPU and memory resources for requests and limits are NOT specified for the container. Pod MUST be able to read the default memory limits from the mounted DownwardAPIVolumeFiles. */ framework.ConformanceIt("should provide node allocatable (memory) as default memory limit if the limit is not set [NodeConformance]", func() { podName := "downwardapi-volume-" + string(uuid.NewUUID()) pod := downwardAPIVolumeForDefaultContainerResources(podName, "/etc/podinfo/memory_limit") f.TestContainerOutputRegexp("downward API volume plugin", pod, 0, []string{"[1-9]"}) }) }) func projectedDownwardAPIVolumePodForModeTest(name, filePath string, itemMode, defaultMode *int32) *v1.Pod { pod := projectedDownwardAPIVolumeBasePod(name, nil, nil) pod.Spec.Containers = []v1.Container{ { Name:    "client-container", Image:   imageutils.GetE2EImage(imageutils.Mounttest), Command: []string{"/mounttest", "--file_mode=" + filePath}, VolumeMounts: []v1.VolumeMount{ { Name:      "podinfo", MountPath: "/etc/podinfo", }, }, }, } if itemMode != nil { pod.Spec.Volumes[0].VolumeSource.Projected.Sources[0].DownwardAPI.Items[0].Mode = itemMode } if defaultMode != nil { pod.Spec.Volumes[0].VolumeSource.Projected.DefaultMode = defaultMode } return pod } func projectedDownwardAPIVolumePodForUpdateTest(name string, labels, annotations map[string]string, filePath string) *v1.Pod { pod := projectedDownwardAPIVolumeBasePod(name, labels, annotations) pod.Spec.Containers = []v1.Container{ { Name:    "client-container", Image:   imageutils.GetE2EImage(imageutils.Mounttest), Command: []string{"/mounttest", "--break_on_expected_content=false", "--retry_time=1200", "--file_content_in_loop=" + filePath}, VolumeMounts: []v1.VolumeMount{ { Name:      "podinfo", MountPath: "/etc/podinfo", ReadOnly:  false, }, }, }, } applyLabelsAndAnnotationsToProjectedDownwardAPIPod(labels, annotations, pod) return pod } func projectedDownwardAPIVolumeBasePod(name string, labels, annotations map[string]string) *v1.Pod { pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:        name, Labels:      labels, Annotations: annotations, }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: "podinfo", VolumeSource: v1.VolumeSource{ Projected: &v1.ProjectedVolumeSource{ Sources: []v1.VolumeProjection{ { DownwardAPI: &v1.DownwardAPIProjection{ Items: []v1.DownwardAPIVolumeFile{ { Path: "podname", FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "metadata.name", }, }, { Path: "cpu_limit", ResourceFieldRef: &v1.ResourceFieldSelector{ ContainerName: "client-container", Resource:      "limits.cpu", }, }, { Path: "cpu_request", ResourceFieldRef: &v1.ResourceFieldSelector{ ContainerName: "client-container", Resource:      "requests.cpu", }, }, { Path: "memory_limit", ResourceFieldRef: &v1.ResourceFieldSelector{ ContainerName: "client-container", Resource:      "limits.memory", }, }, { Path: "memory_request", ResourceFieldRef: &v1.ResourceFieldSelector{ ContainerName: "client-container", Resource:      "requests.memory", }, }, }, }, }, }, }, }, }, }, RestartPolicy: v1.RestartPolicyNever, }, } return pod } func applyLabelsAndAnnotationsToProjectedDownwardAPIPod(labels, annotations map[string]string, pod *v1.Pod) { if len(labels) > 0 { pod.Spec.Volumes[0].VolumeSource.Projected.Sources[0].DownwardAPI.Items = append(pod.Spec.Volumes[0].VolumeSource.Projected.Sources[0].DownwardAPI.Items, v1.DownwardAPIVolumeFile{ Path: "labels", FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "metadata.labels", }, }) } if len(annotations) > 0 { pod.Spec.Volumes[0].VolumeSource.Projected.Sources[0].DownwardAPI.Items = append(pod.Spec.Volumes[0].VolumeSource.Projected.Sources[0].DownwardAPI.Items, v1.DownwardAPIVolumeFile{ Path: "annotations", FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "metadata.annotations", }, }) } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Running the non- conformance tests with produces unpredictable results, sometimes tests show up again with a on the end. This is due to the following tests having multiple tests with identical names: - - What you expected to happen: Tests should have unique names. How to reproduce it (as minimally and precisely as possible): run the conformance tests with a skip on and Anything else we need to know?: example here: https://k8s- /sig node /sig testing cc
er this is probably better /sig storage cc
/assign /assign
/area conformance
/milestone v1.13


Question: } factory.Start(wait.NeverStop) <ins> ctx, cancel := context.WithTimeout(context.Background(), TestInformerSyncTimeout) defer cancel() for ty, ok := range factory.WaitForCacheSync(ctx.Done()) { if !ok { t.Fatalf("failed to sync: %#v", ty) } } </ins> host := volumetest.NewFakeVolumeHostWithCSINodeName(t, tmpDir,

Which test(s) are flaking: TestAttacherWithCSIDriver Reason for failure: This is flaking rarely enough that it is not caught by our CI jobs, which currently tolerate up to 2 unit test failures per run (!). With that toleration removed in , this flake has been seen (). Reproducible with the following steps: a test binary for the affected package: the affected test using the tool (setting the timeout to a generous timeout relative to the normal test runtime of .02 seconds): failures are seen, look at the logs to see the details about the error: /sig storage cc
/assign can you help investigate this flake?
I have found the root cause. /assign
filed to address this issue.


Question: } for _, t := range tests { test := t <del> ginkgo.It(t.name, ginkgo.SpecTimeout(f.Timeouts.PodStart), func(ctx context.Context) { </del> <ins> ginkgo.It(t.name, ginkgo.NodeTimeout(f.Timeouts.PodStart), func(ctx context.Context) { </ins> scName := "mock-csi-storage-capacity-" + f.UniqueName m.init(ctx, testParameters{ registerDriver:  true,

/kind flake /sig storage It seems the timeout is too short or should not apply to the cleanup of the test, since the test is failing because the context is canceled during the timeout
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
See


Question: find-release-tars upload-server-tars <ins> # ensure that environmental variables specifying number of migs to create set_num_migs </ins> if [[ ${KUBE_USE_EXISTING_MASTER:-} == "true" ]]; then create-nodes create-autoscaler

cc:
It was broken by . I'll send a fix in a sec.
Sorry & thanks for fixing.


Question: func (e *eventBroadcasterImpl) StartEventWatcher(eventHandler func(*v1.Event)) watch.Interface { watcher, err := e.Watch() if err != nil { <ins> // This function traditionally returns no error even though it can fail. // Instead, it logs the error and returns an empty watch. The empty // watch ensures that callers don't crash when calling Stop. </ins> klog.FromContext(e.cancelationCtx).Error(err, "Unable start event watcher (will not retry!)") <ins> return watch.NewEmptyWatch() </ins> } go func() { defer utilruntime.HandleCrash()

tainteviction NA No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The panic is in the event watcher, you have landed several changes recently that are most probably related, can you PTAL? /sig instrumentation
ok, so the problem is this The watcher fails to be created but the goroutine keeps going and tries to get a channel from it /assign
That problem has already been there before the recent changes: When constructing the watch fails, that code needs to return instead of spawning the goroutine. I think this is okay: The indicates that the occurs after the test has completed. Before the recent changes, creating the watch would have succeeded and the test would have permanently leaked the goroutines. In that sense, the recent changes work as intended.
ok, so this just uncovered an existing bug, those are the positive news
I don't know if we'll translate the problem to other layer, as will return a nil , it seems StartEventWatcher has to also return an error?
I'm adding -
Because of , this shouldn't occur in production.


Question: // but we use a throttled logger to prevent spamming. globalThrottledLogger.Infof("%s", message) } <del> metrics.RateLimiterLatency.Observe(ctx, r.verb, *r.URL(), latency) </del> <ins> metrics.RateLimiterLatency.Observe(ctx, r.verb, r.finalURLTemplate(), latency) </ins> return err }

Hello, we have been using client-go to measure the latency for all client-go traffic from Cilium to kube-apiserver. We have that since v1.24.0 the URL path is no longer aggregated in a way that this cardinality would no longer happen as presented in the following image: <img width="330" alt="Screen Shot 2022-08-08 at 10 18 24 AM" src="https://user-"It seems this functionality was removed as a part of as it seems that PR only took into account that users of the would only be using it to measure latency to a particular host and not for a particular URL path as we are doing in . Would it be reasonable to revert commit in order to fix this regression? Thanks cc
/sig api-machinery /sig instrumentation
But reverting will bring back the cardinality problem, will not? The latency of each URL path is available in the apiserver metrics ...
We have never any cardinality problem with the previous code since we were not using the "host" field which is what I assume the original issue about cardinality was trying to fix. We only used the "path" of an URL, which was aggregated in a way that would not be facing any cardinality, like the figure attached in this GH description. Now, starting with = 1.24.0, each and is actually a different value which does cause the cardinality problem to appear.
ah, I see the disconnection, I wasn't understanding the issue sorry. The Interface uses the url as parameter to keep backwards compatibility, but the method that gathers the metrics only uses the host to avoid the cardinality problem. You have to update the method that gathers the metrics to only use the host part of the url
Filed
should fix the code in kubernetes ? however, that will break consumer of the Interface that still wants to process the path, but we know using the path will cause issues with cardinality :shrug:
I would be fine with adding a new label to mitigate that issue since originally the cardinality explosion that was reported was caused by the query string and not the path: When trimming down the whole URL to only the host we even had a discussion to keep the path, but we decided not to since we didn't have any use case for this data.

If you have a use case where you need the path in order to get more granularity on the latency of your network then I would be fine with adding the label to Kubernetes, but another path forward could be to also remove the path information from cilium if deemed unnecessary. In the most common scenarios, you will only need the host to get the network latency based on different proxy routes.
yeah we can't really do much about it, it is either we leak the unsanitized paths or we prevent the users from using the path completely. Since we want to not break the interface and keep the URL parameter, we might just have to keep sanitizing the paths for our consumers even though we are not using the path anymore.
IIUIC the RESTClient is used in the dynamic client-go too, for these users with 1900 CRDs, the path is really going to explode for them
Then it seems that the best option to not break users would be to revert , but at the same time I don't really like the idea of spending CPU time sanitizing something that will never be used in the RESTClient
A suggestion would be to create a new interface similar to the that would have the following method: Thus, those users with 1900 CRDs would use the instead of the .
after digging a little bit the underlying issue it looks that the problem was initially caused by a kubelet bug as pointed out . The dependents of client-go don't have access to all the fields of the struct which uses heavily to generate the aggregated URL.
well, it is clear now that current status is wrong, we pass the whole URL to the interface and rely on the to use only the Host part of the URL :smile: I'm fine with whatever you decide, to restore or just completely strip the Path Sorry for the unfortunate consequence, at that time I didn't consider that the Interface/Method drift and thought that function was orphan
/triage accepted /assign


Question: func TestPriorityQueue_Update(t *testing.T) { q := createAndRunPriorityQueue(newDefaultFramework()) q.Update(nil, &highPriorityPod) <ins> q.lock.RLock() </ins> if _, exists, _ := q.activeQ.Get(newPodInfoNoTimestamp(&highPriorityPod)); !exists { t.Errorf("Expected %v to be added to activeQ.", highPriorityPod.Name) } <ins> q.lock.RUnlock() </ins> if len(q.nominatedPods.nominatedPods) != 0 { t.Errorf("Expected nomindatePods to be empty: %v", q.nominatedPods) } // Update highPriorityPod and add a nominatedNodeName to it. q.Update(&highPriorityPod, &highPriNominatedPod) <ins> q.lock.RLock() </ins> if q.activeQ.Len() != 1 { t.Error("Expected only one item in activeQ.") } <ins> q.lock.RUnlock() </ins> if len(q.nominatedPods.nominatedPods) != 1 { t.Errorf("Expected one item in nomindatePods map: %v", q.nominatedPods) } // Updating an unschedulable pod which is not in any of the two queues, should // add the pod to activeQ. q.Update(&unschedulablePod, &unschedulablePod) <ins> q.lock.RLock() </ins> if _, exists, _ := q.activeQ.Get(newPodInfoNoTimestamp(&unschedulablePod)); !exists { t.Errorf("Expected %v to be added to activeQ.", unschedulablePod.Name) } <ins> q.lock.RUnlock() </ins> // Updating a pod that is already in activeQ, should not change it. q.Update(&unschedulablePod, &unschedulablePod) if len(q.unschedulableQ.podInfoMap) != 0 { t.Error("Expected unschedulableQ to be empty.") } <ins> q.lock.RLock() </ins> if _, exists, _ := q.activeQ.Get(newPodInfoNoTimestamp(&unschedulablePod)); !exists { t.Errorf("Expected: %v to be added to activeQ.", unschedulablePod.Name) } <ins> q.lock.RUnlock() </ins> if p, err := q.Pop(); err != nil || p.Pod != &highPriNominatedPod { t.Errorf("Expected: %v after Pop, but got: %v", highPriorityPod.Name, p.Pod.Name) }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: pkg/scheduler/internal/queue Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /assign
I just sent a pr for it according to the error message, but I'm not very sure if it will make things all right.
/assign