Question: <ins> { "kind": "Service", "apiVersion": "v1", "metadata": { "name": "glusterfs-cluster" }, "spec": { "ports": [ {"port": 1} ] } } </ins>

I create a glusterfs endpoint, but after I restart kube-apiserver, the endpoint is missing. Does Kubernetes persistence endpoints?
Do you mean and Endpoints object in the API or something else? , Bin Wang wrote:
Here is what I do: Then use I can see this line: But after I restart kube-apiserver, this endpoint is lost.
Do you have a Service named ? Otherwise it should persist, just the same as any other API object , Bin Wang wrote:
No. This endpoints is just used for persistent volumes, which is used by other pods. I followed . Is it necessary to define a service?
Can one of you guys try to repro?
What does that mean exactly? What kind of cluster are you running? Does anything persist and endure between restarts of the API server? When you restart the API server (why?), are you also restarting etcd or other services?
The cluster just have one master. The pods, secrets, controllers and services are recovered after the restart, but the endpoints are not. About why I restart the API server, well, it may fail (or something else). You cannot expect the api-server runs and never restart. After I restart it, I expect all the things are recovered. In my case, I just restart api-server through /etc/init.d/kube-apiserver (kill the process manually should do the same thing). Etcd is always running and the data in it is not lost.
Any update on this? I cannot use kubernetes in production if this problem is not solved. Or do you have a work around, like define a service with the glusterfs endpoints?
I'm running into this too on GKE so I'm not sure if it's the api-server restarting, but just now after a loadtest and intermittently a couple of other times, after my app tries to come back up it failes to mount from gluster due to the gluster endpoints having mysteriously disappeared.
Do you mean the whole object disappeared or that it exists but is empty?
In the original YAML "IP" should be "ip"
does not show the resource On Mar 27, 2017 8:16 PM, "Tim Hockin" <:
I am trying to repro, but no luck so far.

Just checked.. IP is does come up, and it works well enough as the pods are able to mount the gluster volumes, but every once in a while I find that they're gone. On Mar 27, 2017 8:18 PM, "Tim Hockin" <:
I haven't been able to either, it happens every once in a while. I think maybe they're deleted before I notice and I only notice because a pod is fails and on restart it fails to mount it's volume On Mar 27, 2017 8:21 PM, "Tim Hockin" <:
Just checked a cluster I created an hour ago, and yup all the pods that had been created are running smoothly but the gluster endpoints resource is gone , Jonathan <: — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
Interesting. It appears this is intentional. If you create a service with clusterIP: None and an empty selector, this will not be activated. It is trying to prevent submarine problems, such as creating an Endpoints then later creating a Service and conflicting. , Jonathan Kosgei <:
So the fix is to create a corresponding service for the endpoints?
Yes. Endpoints are usually a derived value, and in hindsight they should probably have been a sub-resource of Service. On Mar 28, 2017 7:20 AM, "Jonathan Kosgei" <:
In case you're wondering (like I was) how to attach a service to the EP the was updated to include an example service for it.
I'm having this issue again in a very different setup, basically I'm using the Nginx Ingress controller and it uses endpoints and these endpoints keep disappearing intermittently breaking all ingress to the cluster. Opened an issue at the nginx-ingress repo Saving the endpoints with a and recreating them when resolution failure occurs fixes it immediately.
this will make kubelet notready if i create a Service named glusterfs-cluster, after reboot endpoint is not lost. but the kubelet will be notready , something wrong? hope response tks.
The correct explanation should be When the cluster starts Endpoint-Controller Will check all Endpoint list Then put Endpoint name in workqueue, worker get value from workqueue, Found Endpoint no corresponding Service,Just delete Endpoint.

yes, the manually created EP resource will not be reconciled by the controller. This is the cause. Usually we do not create EP, this kind of secondary resource directly. It is created for the corresponding service resource.


Question: serviceUpdateResult := proxier.serviceMap.Update(proxier.serviceChanges) endpointUpdateResult := proxier.endpointsMap.Update(proxier.endpointsChanges) <del> staleServices := serviceUpdateResult.UDPStaleClusterIP </del> <ins> // We need to detect stale connections to UDP Services so we // can clean dangling conntrack entries that can blackhole traffic. conntrackCleanupServiceIPs := serviceUpdateResult.UDPStaleClusterIP conntrackCleanupServiceNodePorts := sets.NewInt() </ins> // merge stale services gathered from updateEndpointsMap <ins> // an UDP service that changes from 0 to non-0 endpoints is considered stale. </ins> for _, svcPortName := range endpointUpdateResult.StaleServiceNames { if svcInfo, ok := proxier.serviceMap[svcPortName]; ok && svcInfo != nil && conntrack.IsClearConntrackNeeded(svcInfo.Protocol()) { klog.V(2).InfoS("Stale service", "protocol", strings.ToLower(string(svcInfo.Protocol())), "svcPortName", svcPortName.String(), "clusterIP", svcInfo.ClusterIP().String()) <del> staleServices.Insert(svcInfo.ClusterIP().String()) </del> <ins> conntrackCleanupServiceIPs.Insert(svcInfo.ClusterIP().String()) </ins> for _, extIP := range svcInfo.ExternalIPStrings() { <del> staleServices.Insert(extIP) </del> <ins> conntrackCleanupServiceIPs.Insert(extIP) } nodePort := svcInfo.NodePort() if svcInfo.Protocol() == v1.ProtocolUDP && nodePort != 0 { klog.V(2).Infof("Stale %s service NodePort %v -> %d", strings.ToLower(string(svcInfo.Protocol())), svcPortName, nodePort) conntrackCleanupServiceNodePorts.Insert(nodePort) </ins> } } }

Which jobs are flaking: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are flaking: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service Testgrid link: https://k8s- Reason for failure: /sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
CRI-O CI is also hitting this failure. We have been unable to reproduce locally. our CI runs on aws
hmm, I only see 1 failure out of 4 and seems that the connectivity didn't work at all
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: JacobTanenbaum. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Saw the test flaking recently: Wondering if this is just a timeout issue.

Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.

I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.

After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?

3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.

so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.

Would love to help out in reproing this, I'm at a loss for good next step so any suggestions welcome!
that's the problem with these flakes, they are not easy to repro, and with this low rate of failures it may be caused for combination of running tests, environmental problems (cpu, memory, ...) , ... I always start for checking the current failures and try to find patterns
Let's take a look to one occurrence The tests fails because can't connect to the first pod If we check the code of the tests, we can see that it always uses the port as source port, we can use that to identify the nodePort assigned interesting thing here, we are source nating the pod ip, and we don't see any other conntrack entry The test verifies it has endpoints however, kube-proxy is using endpointlsices since this version but it has no entries so the iptables rules to forward the traffic are not there Seems we have to adapt this function in the e2e framework to use endpointslices cc: EDIT seems that this function is being used by all the network tests that can cause flakes if the endpoint is updated before the endpointslice (we switched recently kube-proxy to use the later)
Great catch. Ideally this function should validate if both Endpoint and appropriate EndpointSlice objects(s) exist.
Actually - let me take it and send a PR for it.
is out for review
Interesting failures here It fails to connect to the first pod
Hmm - I took a quick look at kube-proxy logs and these look fine to me....
I clicked too fast, sorry ... I was looking at the wrong service and I though I've found something
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.

just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job:


Question: CgroupManagerOperationsKey   = "cgroup_manager_latency_microseconds" PodWorkerStartLatencyKey     = "pod_worker_start_latency_microseconds" PLEGRelistLatencyKey         = "pleg_relist_latency_microseconds" <ins> PLEGDiscardEventsKey         = "pleg_discard_events" </ins> PLEGRelistIntervalKey        = "pleg_relist_interval_microseconds" EvictionStatsAgeKey          = "eviction_stats_age_microseconds" VolumeStatsCapacityBytesKey  = "volume_stats_capacity_bytes"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In our stress testing, we found the deadlock issue in PLEG relist() for healthy checking and Kubelet syncLoop() The goroutine stack trace is as following We did some investigation, and found the following issue. The main loop for event processing in Kubelet will check if there are some runtimeErrors: if yes, wait until runtime back to normal; if no, it will call syncLoopIteration to read and process events. And the runtime healthChecks depends on the But in the relist() impl, it will send the pod lifecycle event to eventChannel which may introduce deadlock when eventChannel is full. And in the above code, the eventChannel will not be consumed if the runtime is not healthy. In some situation, the eventChannel may full, and the relist() will hang in sending new pod lifecycle event. And the runtime status will not be able to recovered. It is related to issues What you expected to happen: No such issue How to reproduce it (as minimally and precisely as possible): It is not easy to reproduce, it happens after few days stress testing (scaling the deployment repeatedly with high load). Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): CentOS 7.4 Kernel (e.g. ): Linux bjc-qtt-k8s-node-010 3.10.0-693.2.2.el7.x8664 SMP Tue Sep 12 22:26:13 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig node
Sure, thank you We had some internal fix to break the deadlock, but I am wondering if there are good fix for such kind of issue.
When pleg do relist, can we invoke to judge runtime is ok, if not, not do relist
I submit a PR
Thanks for finding the issue! The issue description makes sense to me. I think PLEG is a core component, we should try our best to unblock it.

The event channel becoming full seems more like an issue in the event consumer side, which shouldn't block PLEG. One solution is to queue the events in PLEG and send later, but that doesn't fundamentally solve the problem, if we don't set a limit on the event number we queue, it seems like a potential memory leak to me; if we set a limit, we may eventually hit this problem again. Another solution is to add a retry for sending event in PLEG, and set a bit in PLEG in between the retry which tells that PLEG is not blocking, it is trying to send out the event. And in the function of PLEG, it returns healthy if the bit is set. Actually, even if we fail to send the event and discard it, since we have already updated the pod cache, the periodical pod resync should eventually pick it up. This may delay pod sync for up to . But if the channel is full, it means that either 1) the syncloop is blocked for some reason, or 2) the node is under heavy loads. In both cases, even the event is successfully sent, the pod sync will likely be delayed anyhow. We can: 1) Enlarge the event channel size to reduce the chance that the channel becoming full. Currently it is arbitrarily set as . 2) Discard the event in PLEG if the channel is already full. In the 3 options above, I'm fine with either 2 or 3. Option 2 doesn't have the 1min pod sync delay, but a bit ad-hoc; option 3 is more generic, but may delay pod sync. There might be other options. /cc Any thoughts?
If PLEG is already blocked by a full event channel, this doesn't help right?
cc/ This looks like the one production issue you mentioned to me a while back.
of options enumerated, option 3 was preferred for me.
Thanks for reporting the issue! what was the load on the node that triggered this condition? If the node wasn't function at all, and the events slowly built up over time, bumping the size of the channel won't help. This should work, and doesn't complicate the logic in PLEG more, so it's my preferred option too.

We need some unit tests, and ideally, a stress test, as mentioned in the sig-node meeting.
Agree, giving a proper size for event channel is tricky as the scale also grows with time. We will ship fix as option 3 () for 1.14. Thanks for summarizing all these up as sig meeting! /assign
If possible, could you share in how large the stress is? It helps understand the issue and kubelet stress handling more. :)
Sure, our product team ( ) will input the stress test data after removing sensitive information in follow up comments.
Thanks for reply. Following is our testcase to reproduce this issue: The test cluster has 5 node on aliyun ECS(like EC2). Three 4C8G machine as master node, Two 96C384G as worker node. Firstly, We create 2 deployment running on the worker node. One deployment is http server pods. Another is siege pods to produce network stress to http server deployment. Those stress test deployments produce 50 CPU Core load on worker node. Then we create a job frequently create/delete 20 pod every 5 seconds. After about 1 day of stess test, The worker node will become NotReady. Dig the pprof goroutine stacktrace found the channel block in .
Yeah, I can see how frequent container state change can cause this issue. The number is very useful, thanks a lot!
cloud you please help to share how to dump the goroutines stack trace of kubelet like what you pasted in your comment. thanks a lot.
This issue is fixed by
using pprof is a answer to my question thank you anyway.


Question: resources.CpuPeriod = cpuPeriod } <ins> // runc requires cgroupv2 for unified mode if libcontainercgroups.IsCgroup2UnifiedMode() { resources.Unified = map[string]string{ // Ask the kernel to kill all processes in the container cgroup in case of OOM. // See memory.oom.group in https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html for // more info. "memory.oom.group": "1", } } </ins> return &resources }

Follow up on meeting of 28-03-2022 - In the meeting was specified that this most likely won't need a KEP and should be started with a Issue I see that the kernel in cgroups v2 has an option which would make all the pids of a given container to get killed on OOM: Some workloads that fork the process in the container, when the OOM kicks in, it kills only one process of the cgroup, causing a lot of noise due to process failures but health checks being successfull. We think that for our scenarios, killing all the pids in the containter would be better.
/cc
The way I’d summarize this issue with is: A small number of important workloads handle processes and OOM correctly (nginx, mature multi process systems like DBs) - we should not prevent these workloads from benefitting from per process OOM. A much larger and more diverse set of workloads are unlikely to handle OOM of multi process behavior incorrectly (bash child processes, especially in entrypoint scripts) - these users are unlikely to test OOM behavior in advance and thus are much more vulnerable to behaving incorrectly if we don’t default to killing the whole container. That also apples to containers with shared process namespaces. We generally bias towards the later group (low expertise / high diversity / low frequency / unlikely to read documentation) when designing our pod semantics, while allowing advanced users to set optional flags.
/cc
/cc
/sig node
/triage accepted /priority important-longterm


Question: if !ok { return nil, false, nil } <del> if user.password != password { </del> <ins> if subtle.ConstantTimeCompare([]byte(user.password), []byte(password)) == 0 { </ins> return nil, false, nil } return &authenticator.Response{User: user.info}, true, nil

This issue was reported in the Description The kube-apiserver includes multiple authentication backends for client request processing. These range in strength from client certificate authentication to simple HTTP Basic Authentication. When using a password (Basic Authentication), the kube-apiserver does not perform a secure comparison of secret values. In theory, this could allow an attacker to perform a timing attack on the comparison. For example, the AuthenticatePassword function simply performs string comparison to authenticate a user’s password (Figure 9.1). Figure 9.1: Username and password authentication handling in Exploit Scenario Alice runs a Kubernetes cluster in production. In order to support multiple organizational “customers,” she configures the cluster with HTTP Basic Authentication. Eve has a large number of username and password pairs for the organization, and uses the side-channel information from string comparison to tune her credential-stuffing attacks. Recommendation Short term, use a safe, constant-time comparison function such as . The comparison function should take the same amount of time regardless of matching prefix data within the password. Long term, deprecate Basic Authentication in favor of more robust and secure options. Add documentation noting that any Basic Authentication is for use only in development scenarios, and not appropriate for production deployments. This will help users create a robust and secure default stance for all deployments. Anything else we need to know?: See for current status of all issues created from these findings. The vendor gave this issue an ID of ATR-K8S-002 and it was finding 18 of the report. The vendor considers this issue Medium Severity. To view the original finding, begin on page 55 of the Environment: Kubernetes version: 1.13.4 Fixed in v1.16.0 by
/sig auth
/area security /kind bug /remove-kind feature


Question: func (m *kubeGenericRuntimeManager) killContainer(pod *v1.Pod, containerID kubecontainer.ContainerID, containerName string, reason string, gracePeriodOverride *int64) error { var containerSpec *v1.Container if pod != nil { <del> containerSpec = kubecontainer.GetContainerSpec(pod, containerName) </del> <ins> if containerSpec = kubecontainer.GetContainerSpec(pod, containerName); containerSpec == nil { return fmt.Errorf("failed to get containerSpec %q(id=%q) in pod %q when killing container for reason %q", containerName, containerID.String(), format.Pod(pod), reason) } </ins> } else { // Restore necessary information if one of the specs is nil. restoredPod, restoredContainer, err := m.restoreSpecsFromContainerLabels(containerID)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I tried to use virtlet with Kubernetes release 1.6.6, but a pod with a VM in it gets stuck in "ContainerCreating" state. This problem does not happen with Kubernetes release 1.6.3. The log of the responsible kubelet shows a panic when trying to restart the container. I do not know why the container is being restarted, but that should not cause a panic in any case. Here is an excerpt from the kubelet's log: It is easy to see a code path that can cause this panic. The offending line in is this: and there is indeed a way for to be nil at this point. Earlier in that function we see the line and there is no intervening statement that checks whether is ; ends with . What you expected to happen: I expected the virtlet to run, like it does with Kubernetes release 1.6.3. How to reproduce it (as minimally and precisely as possible): That would take some work, I hope the diagnosis above makes this moot. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: "bare metal" (really VMs, but who cares?) OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS Kernel (e.g. ): Linux n11rdu1ctrl 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: I installed Kubernetes and virtlet using some local Ansible Others: I am using a local fork of virtlet
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : </details
, did you do any update action against that Pods?

I upped the kubelet's to 4, and got an interesting clue when I tried the experiment again. Here is an excerpt from the more verbose log. Note the empty container name in the "Killing unwanted container" entry. That telling log entry comes from these lines in which is soon followed by .
I only created the pod, no updates.
Is there a running container with id something like when the panic occurs? A quick fix is to add checking whether containerSpec is nil as you mentioned. But it's weird that gets a container without name.
Did virtlet set correctly? Seems it is "", which caused the problem.
Create a PR () that checking containerSpec to avoid panic. And it seems there's no UT case for , I'll create related PRs for
The actual Docker containers are all gone now. But here is what I got from shortly before opening this issue: I will look into the question asked.


Question: } } <ins> func addNotReadyPodsWithSpecifiedRestartPolicyAndPhase(store cache.Store, namespace string, nPods int, nPorts int, restartPolicy v1.RestartPolicy, podPhase v1.PodPhase) { for i := 0; i < nPods; i++ { p := &v1.Pod{ TypeMeta: metav1.TypeMeta{APIVersion: api.Registry.GroupOrDie(v1.GroupName).GroupVersion.String()}, ObjectMeta: metav1.ObjectMeta{ Namespace: namespace, Name:      fmt.Sprintf("pod%d", i), Labels:    map[string]string{"foo": "bar"}, }, Spec: v1.PodSpec{ RestartPolicy: restartPolicy, Containers:    []v1.Container{{Ports: []v1.ContainerPort{}}}, }, Status: v1.PodStatus{ PodIP: fmt.Sprintf("1.2.3.%d", 4+i), Phase: podPhase, Conditions: []v1.PodCondition{ { Type:   v1.PodReady, Status: v1.ConditionFalse, }, }, }, } for j := 0; j < nPorts; j++ { p.Spec.Containers[0].Ports = append(p.Spec.Containers[0].Ports, v1.ContainerPort{Name: fmt.Sprintf("port%d", i), ContainerPort: int32(8080 + j)}) } store.Add(p) } } </ins> type serverResponse struct { statusCode int obj        interface{}

If you create a job in which pods are crashing and a service spawning those pods, this may result in generating extremely huge Endpoints object (which will cause different performance issues). [Based on situation from one of our customers cluster]. After 30m from creating the objects below in my 3 node-cluster I have an endpoint which has almost 1MB of size: Example objects which are enough to cause this : :
- I think that we want to have "failed pods" that have "restart policy" "Always" to be part of "NotReadyAddresses". But do we want to keep pod that are in Failed state and have restart policy "Never"? I think it doesn't make sense. And this would be a quick and easy fix to this particular problem...
Can you update the issue title to "Failed pods with restart policy=Never should not be in the Endpoints.NotReadyAddresses"
done Do you agree that this is a reasonable request?
Geniunely interested: in what case do you want a service and a job? cc:
I can imagine a map-reduce-like thing that serves status from each worker, and uses a headless Service for discovery.
Yes - this is the example I've heard during discussions on last Kubecon.
Well - I don't know if it was map-reduce-like-thing, but it was about headless Service for discovery (and also reporting status from workers).
/assign
GitHub didn't allow me to assign the following users: xiangpengzhao. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We should also not put such pods in the Endpoints.NotReadyAddresses: Succeeded pods with restart policy=Never Succeeded pods with restart policy=OnFailure
- thanks for working on it. I commented on the PR.


Question: iptables  ethtool  ca-certificates  <ins> conntrack  </ins> file  util-linux  socat

requires the binary but the binary is not available in . is provided by the Debian package.
It seems the problem has been introduced with PR : the file is missing in the command
Thank you for the fix! Is this planned for the next 1.2.x release?
I'll be fixed with the upcoming release.


Question: Type=oneshot RemainAfterExit=yes ExecStartPre=/usr/bin/mkdir -p /opt/cni <del> ExecStartPre=/usr/bin/curl --fail --silent --show-error --location --create-dirs --output /opt/downloads/cni.tar.gz https://storage.googleapis.com/kubernetes-release/network-plugins/cni-8a936732094c0941e1543ef5d292a1f4fffa1ac5.tar.gz </del> <ins> ExecStartPre=/usr/bin/curl --fail --silent --show-error --location --create-dirs --output /opt/downloads/cni.tar.gz https://storage.googleapis.com/kubernetes-release/network-plugins/cni-9d5e6e60e79491207834ae8439e80c943db65a69.tar.gz </ins> ExecStart=/usr/bin/tar xf /opt/downloads/cni.tar.gz -C /opt/cni/ - name: kubernetes-install-docker2aci.service

We should update cni binaries to containernetworking/cni and release binaries for all arches. I do not have push access to the repo, so I guess some googler have to do it.
Thanks. I have this on my plate.


Question: "k8s.io/kubernetes/pkg/controller/endpoint" "k8s.io/kubernetes/test/integration/framework" "k8s.io/kubernetes/test/utils/ktesting" <ins> netutils "k8s.io/utils/net" </ins> ) func TestEndpointUpdates(t *testing.T) {

This issue was supposed to have fixed the issue where endpoint stay out of sync I also wrote a small script which would get me the out of sync endpoints compared to the endpointslices I expect the endpoints to eventually sync and reflect the most upto date information. I have just deployed the newer patch to our cluster and that has resulted in endpoints never ending up being updated if the status goes out of sync. No response Client Version: v1.29.7 Kustomize Version: v5.0.4-0.- Server Version: v1.29.7 <details</detailsalmalinux-9 <details</detailscri-o No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
from the logs, endpoint controller was retrying test1/test-qa in the expected way. It has retried 5 times and the next retry should be after 320ms, which should long enough for the informer cache to get the updated endpoints. If there is no more logs related to the failure, I think it may have synced it successfully. The python script used to detect the failure seems capturing diff between endpoints and endpointslice at a moment, but that could happen if there is any Pod change in the cluster, and technically the script is comparing endpoints of one moment with endpointslice of another moment. So did you see the endpoints not being synced after endpoint controller stopped retrying it?
Yeah i still see it and see similar messages continuously. Let me post some additional log messages after the one i posted above.
Observed the same issue in 1.29 and 1.30. I keep seeing the same error then new pods were behind a service. And the endpoint never catch up after all pods are settled.
could you provide complete log of kube-controller-manager and the yamls of the related endpoints and endpointslice?
/assign Are you able to triage this since you are already looking at this?

Yes i can get that over by EOD today
Sure, I'm waiting for more logs and the content of the endpoints to understand how it happened. Currently there are only retry logs which is expected in some cases.
We found a customer that triggered an edge case in your change of , which will cause Endpoints never updated. They have a mutatingWebhook on Endpoints object that automatically injects a label, let's say . syncServices, the endpoints controller , and they won't match since Endpoints have the extra "dummy-k: value". endpoints controller will by removing the label. mutatingWebhook will automatically adds back the label as part of the update request. From the APIServer's POV, the Endpoints object don't have any changes, thus the update request will succeed but returns the same resourceVersion as before. change in will mark that resourceVersion as stale. future pods events won't be handled correctly due to the step didn't result in a new resourceVersion. I think we should fix this by check the resulting resourceVersion from the update request, and only mark the old resourceVersion as stale when a new resourceVersion is generated. Not sure whether this is the same case others in this thread encountered :D
thanks for explaining the use case and the analysis. I did analyze the code after the issue was opened and suspected the only possibility is the returned resource version somehow equals to the one in the request but didn't know how it could happen after it has checked the desired endpoints and the current endpoints are different: Now I understand this could happen when the endpoints appear different on client side but are mutated to be same on server side. Do you plan to create a PR since you have proposed a fix? could you confirm if this is also your case, i.e. there is a mutatingWebhook for endpoints objects that could make the endpoints different from what endpoints controller expects?
Yeah, i can do a fix for this
- can we assign to you?
Sure, I'm already assigned. Given that we've had three user reports on this issue, I think it's important to address it as soon as possible. Could you please let me know when you plan to create the PR? If you're currently occupied, I'm happy to handle it and submit the PR.
Sorry, were occupied by some internal incidents last week. Just created a WIP PR for it.

Will test it in kops for this specific webhook scenario in the following two days and and remove the WIP flag
/assign


Question: // Create a few pods for tests. pods := []*v1.Pod{} <del> for i := 0; i < 10; i++ { </del> <ins> for i := 0; i < 20; i++ { </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      fmt.Sprintf("test-pod%v", i),

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If we delete node first, then remove pods on this node, the scheduler will crash in function What you expected to happen: Not crashed How to reproduce it (as minimally and precisely as possible): some pods on node A delete node A delete pods on node A Anything else we need to know?: Environment: Kubernetes version (use ): 1.16-1.18, even master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
can you show the core dump please of this crash?
/assign
I patched the unit test shown in into the , and branches. None crashed. I'll await for the crash dump.
Here is the fresh dump.
and which version was this?
This is master branch. And I've tested it from 1.16-1.18, it can be reproduced easily.
Thanks a lot. I was able to finally reproduce in unit tests. The scenario is: a node with at least 2 pods. node. pod1. In unit test, this is first an update to set the termination timestamp. No issues up to this point. pod2. Again, first an update to set the termination timestamp. Here it crashes. I'm working on a fix.
It looks like this is not an issue in 1.17 and below, because we preserved a placeholder node where the operations occur. do you have stacktraces for 1.17 and 1.16?
Yes, it seems only be found at 1.18. :smile:


Question: // If pod exists, reset remountRequired value podObj.remountRequired = false podObj.volumeMountStateForPod = markVolumeOpts.VolumeMountState <ins> if mounter != nil { // The mounter stored in the object may have old information, // use the newest one. podObj.mounter = mounter } </ins> asw.attachedVolumes[volumeName].mountedPods[podName] = podObj return nil }

Using a CSI volume on a Linux machine with SELinux in enforcing mode. I noticed when the volume is used in a pod and the first NodePublishVolume for that pod fails (e.g. timeout), the volume does not get correct SELinux labels. The volume gets correct SELinux labels. With a hacked CSI driver that returns error on the first NodePublishVolume. a pod with a CSI volume. the first NodePublishVolume fails (e.g. DeadlineExceeded). that a subsequent NodePublishVolume succeeds. the pod is Running. SELinux label of the volume Got , where was expected. Kubernetes version (use ): 1.20.something Cloud provider or hardware configuration: N/A OS (e.g: ): "Red Hat Enterprise Linux CoreOS 47.83.-0 (Ootpa)"
This breaks applications on clusters with CSI drivers that tend to time out with the first NodePublishVolume - the pods can't access their data and they get instead. /priority important-soon /sig storage /triage accepted
Root cause: VolumeManager's ActualStateOfWorld (ASW) uses a wrong when a volume gets mounted. volume plugin stores a flag if a volume supports SELinux after NodePublish succeeds: the first NodePublish does not succeed, of the stays . This first is used as interface when marking the volume as uncertain in ASW / : is called again. It succeeds and is set correctly to , however, it's a different than the one saved in ASW. The Mounter is ASW is not updated at this point. calls on the from step 2., which returns and the volume is not relabeled.
There are several ways out of this. For example, operation generator should use the same Mounter all the time, even in subsequent attempts. IMO the best would be to modify to update on every call.
Another idea: we could move detection of SELinux on a volume into call. This way, we would not need to pass a flag in around. But the detection can fail with an error and can't return one (at least today).
Implemented this approach in


Question: import ( "encoding/json" "fmt" <ins> "net" </ins> goruntime "runtime" "sort" "strconv"

BUG REPORT: /kind bug What happened: After upgrading the kubernetes cluster from 1.6.4 with the VSphere Cloud Provider enabled to 1.6.5/1.6.6 or 1.6.7, the cluster nodes don't have anymore InternalIPs Several issues occur at this step. For example, it breaks heapster with such an error : What you expected to happen: The node should have an InternalIP. How to reproduce it (as minimally and precisely as possible): Deploy a 1 node kubernetes cluster in a VM, and enable the vsphere cloud provider. Anything else we need to know?: with kubernetes 1.6.4 we get : Downgrading to 1.6.4 fixes the issue. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vsphere OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: It may be related to some changes around PR kubernetes /area platform/vsphere cloudprovider
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cluster-ops /sig network /area platform/vsphere /area cloudprovider
Performing some more tests on a mini instance outside of the production cluster, I think I've identified the issue. The configuration required to start kubelet with the option (otherwise the wrong IP/interface is used to communicate between nodes). The issue is triggered when this option is set. The cloud provider looks OK. But the issue is located here : When the is not , the loop iterates on the node addresses and returns only the first one found. Since PR kubernetes, the list always returns an ExternalIP first. I think the loop in should iterate on all the addresses to rebuild the list, so that it will include the ExternalIP and the InternalIP. Can someone confirm the analysis ?
/sig node
ping
Modifying the code like this solved the issue : which returns : I guess it can be done in a cleaner way, but it was to quickly test the fix.

CC:
Thanks for filing this issue related to vSphere cloud provider. I agree with your analysis. It seems the current function assumes that the IP address specified by user can only be InternalIP. Your fix to is a better solution than work-around in cloud provider.
Hi, what are the next steps to discuss about the PR ? Are you able to make some tests ?


Question: Exec: &api.ExecAction{Command: []string{"touch", file}}, }, } <del> testPodWithHook(podWithHook) </del> <ins> testPodWithExecHook(podWithHook) </ins> }) }) Context("when it is http hook", func() { var targetIP string <del> BeforeEach(func() { By("cleanup the container to handle the HTTPGet hook request.") podHandleHookRequest := getLifecycleHookTestPod("pod-handle-http-request", []string{"sh", "-c", // Create test file when receive request on 1234. fmt.Sprintf("echo -e "HTTP/1.1 200 OKn" | nc -l -p 1234; touch %s", file), }, ) podHandleHookRequest.Spec.Containers[0].Ports = []api.ContainerPort{ { ContainerPort: 1234, Protocol:      api.ProtocolTCP, </del> <ins> podHandleHookRequest := &api.Pod{ ObjectMeta: api.ObjectMeta{ Name: "pod-handle-http-request", }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  "pod-handle-http-request", Image: "gcr.io/google_containers/netexec:1.7", Ports: []api.ContainerPort{ { ContainerPort: 8080, Protocol:      api.ProtocolTCP, }, }, }, </ins> }, <del> } podHandleHookRequest = podClient.CreateSync(podHandleHookRequest) targetIP = podHandleHookRequest.Status.PodIP </del> <ins> }, } BeforeEach(func() { By("create the container to handle the HTTPGet hook request.") newPod := podClient.CreateSync(podHandleHookRequest) targetIP = newPod.Status.PodIP </ins> }) <ins> testPodWithHttpHook := func(podWithHook *api.Pod) { By("create the pod with lifecycle hook") podClient.CreateSync(podWithHook) if podWithHook.Spec.Containers[0].Lifecycle.PostStart != nil { By("check poststart hook") Eventually(func() error { return podClient.MatchContainerOutput(podHandleHookRequest.Name, podHandleHookRequest.Spec.Containers[0].Name, `GET /echo?msg=poststart`) }, postStartWaitTimeout, podCheckInterval).Should(BeNil()) } By("delete the pod with lifecycle hook") podClient.DeleteSync(podWithHook.Name, api.NewDeleteOptions(15), podWaitTimeout) if podWithHook.Spec.Containers[0].Lifecycle.PreStop != nil { By("check prestop hook") Eventually(func() error { return podClient.MatchContainerOutput(podHandleHookRequest.Name, podHandleHookRequest.Spec.Containers[0].Name, `GET /echo?msg=prestop`) }, preStopWaitTimeout, podCheckInterval).Should(BeNil()) } } </ins> It("should execute poststart http hook properly [Conformance]", func() { <del> podWithHook := getLifecycleHookTestPod("pod-with-poststart-http-hook", // Block forever []string{"tail", "-f", "/dev/null"}, ) podWithHook.Spec.Containers[0].Lifecycle = &api.Lifecycle{ PostStart: &api.Handler{ HTTPGet: &api.HTTPGetAction{ Host: targetIP, Port: intstr.FromInt(1234), </del> <ins> podWithHook := &api.Pod{ ObjectMeta: api.ObjectMeta{ Name: "pod-with-poststart-http-hook", }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  "pod-with-poststart-http-hook", Image: framework.GetPauseImageNameForHostArch(), Lifecycle: &api.Lifecycle{ PostStart: &api.Handler{ HTTPGet: &api.HTTPGetAction{ Path: "/echo?msg=poststart", Host: targetIP, Port: intstr.FromInt(8080), }, }, }, }, </ins> }, }, } <del> testPodWithHook(podWithHook) </del> <ins> testPodWithHttpHook(podWithHook) </ins> }) It("should execute prestop http hook properly [Conformance]", func() { <del> podWithHook := getLifecycleHookTestPod("pod-with-prestop-http-hook", // Block forever []string{"tail", "-f", "/dev/null"}, ) podWithHook.Spec.Containers[0].Lifecycle = &api.Lifecycle{ PreStop: &api.Handler{ HTTPGet: &api.HTTPGetAction{ Host: targetIP, Port: intstr.FromInt(1234), </del> <ins> podWithHook := &api.Pod{ ObjectMeta: api.ObjectMeta{ Name: "pod-with-prestop-http-hook", }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  "pod-with-prestop-http-hook", Image: framework.GetPauseImageNameForHostArch(), Lifecycle: &api.Lifecycle{ PreStop: &api.Handler{ HTTPGet: &api.HTTPGetAction{ Path: "/echo?msg=prestop", Host: targetIP, Port: intstr.FromInt(8080), }, }, }, }, </ins> }, }, } <del> testPodWithHook(podWithHook) </del> <ins> testPodWithHttpHook(podWithHook) </ins> }) }) }) }) <del> func getLifecycleHookTestPod(name string, cmd []string) *api.Pod { </del> <ins> func getExecHookTestPod(name string, cmd []string) *api.Pod { </ins> return &api.Pod{ ObjectMeta: api.ObjectMeta{ Name: name,

https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.

[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
the for this issue looks pretty good. We could probably close this issue.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Looks like more recent flakes:
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Adding labels for team node as is failure is part of the node e2e tests. All issues must be labeled either release blocker or non release blocking by end of day 18 November 2016 PST. Given the test flakes very infrequently and doesn't seem to be associated with a particular code change Could either or someone from please advise as to whether you believe this issue is actually a release blocker and I'll update the issue. Any updates from you would also be greatly appreciated.Thanks! cc:
The test was introduced by which causes some flakiness: Re-assign this to Lantao, I think the issue should be in the test itself, and non-resource-blocker. But I temporarily marked it as release blocker after your triage. Thanks!

This is a issue in the test itself. I've sent a fix for this
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: printLabelsMultiline(w, "Labels", pod.Labels) printAnnotationsMultiline(w, "Annotations", pod.Annotations) if pod.DeletionTimestamp != nil { <del> w.Write(LEVEL_0, "Status:tTerminating (lasts %s)n", translateTimestampUntil(*pod.DeletionTimestamp)) </del> <ins> w.Write(LEVEL_0, "Status:tTerminating (lasts %s)n", translateTimestampSince(*pod.DeletionTimestamp)) </ins> w.Write(LEVEL_0, "Termination Grace Period:t%dsn", *pod.DeletionGracePeriodSeconds) } else { w.Write(LEVEL_0, "Status:t%sn", string(pod.Status.Phase))

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubectl describe pv xxx shows last<invalidif pv is terminating ! I found pvc and pod have the same issue What you expected to happen: Terminating (lasts 10s) How to reproduce it (as minimally and precisely as possible): 1 create a pv, pvc or pod is terminating 2 using kubectk describe xxx Anything else we need to know?: Environment: Kubernetes version (use ): 1.13.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig cli
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: if s.isPostStartHookRegistered(priorityAndFairnessConfigConsumerHookName) { } else if c.FlowControl != nil { err := s.AddPostStartHook(priorityAndFairnessConfigConsumerHookName, func(context PostStartHookContext) error { <ins> go c.FlowControl.MaintainObservations(context.StopCh) </ins> go c.FlowControl.Run(context.StopCh) return nil })

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Not observed yet, but deduced from analyzing the code. If a k/apiserver/pkg/util/flowcontrol/fairqueuing.QueueSet, whose metrics.TimedObserverPair is a pair of k/apiserver/pkg/util/flowcontrol/metrics.sampleAndWaterMarkHistograms, gets zero activity for a long while and then gets some activity, then the loop at will go through a lot of iterations, and do so with a lock held. What you expected to happen: No big hiccups in any scenario. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In reviewing I noticed an incorrect fake implementation of Interface. I looked at the comment on the Handle method, and noticed that it does not say when/how the is called. What you expected to happen: I expected the parameter to be as well documented as the others. How to reproduce it (as minimally and precisely as possible): Look at and see how it does not say when/whether/with-what-parameter-values is called. Anything else we need to know?: Environment: Kubernetes version (use ): current Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Any chance we could write a test for the interface which verifies correct plumbing?
/cc
: that should be do-able


Question: <ins> /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package serving import ( "testing" "k8s.io/kubernetes/test/integration/framework" ) func TestMain(m *testing.M) { framework.EtcdMain(m.Run) } </ins>

Quoted from 's Splitting healthz/ into livez/ and readyz/
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/sig scheduling
I read all the content of the PR comment and am willing to work on this issue Do we need to continue some discussion for this issue
Hi , thanks for offering your help. But I was working in progress before knowing you can work on this. However, please feel free to contribute the similar issue
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: return } <ins> allVolumesAdded := true </ins> // Process volume spec for each volume defined in pod for _, podVolume := range pod.Spec.Volumes { volumeSpec, volumeGidValue, err :=

Deployment/Service manifest: logs from events: In aws, the volume is attached, On the box: In only one is mounted: Where can I debug this further?
Take a look at the on your master for events related to your volume . That will show you when the volume was attached/detached. It appears that your volume attached without issue so there should be nothing unusual here. Then take a look at on the node that the volume is supposed to be attached to. Looks for the UID of your pod (). I suspect one of your mounts is failing. It looks like you are using hyperkube so it might be a mount namespace propagation issue.
I believe this happens when the RC/Pod is being created shortly after the PV or PVC. The latter two appear to need several seconds to be fully available (even if the status is already set to "Bound"). I ran into the same issue with a NFS PV. When I delete and create PV+PVC+SVC+RC all in one go, the associated Pod will be in ContainerCreating state for several seconds (10,<60), then log the warning event and afterwards be created without any problems or further delays. When I first create PV+PVC, wait for several seconds (30) and only then create SVC+RC, the pod will be created immediately and no warning event will be logged. One oddity I noticed is that the PVC will have capacity 0 and empty access modes (as shown by ), even though it was setup to request ROX for 1Gi and the PV provides 1Gi with ROX. This might be a different bug, though.
Thanks for digging into it The capacity 0 issue () was fixed by PR and is in v1.3.5. This is more or less expected behavior. The mount logic has exponential backoff so if it rapidly encounters PVC/PV does not exist, it'll back off the retries resulting in a longer mount time. We can consider optimizing this particular scenario. But I'll lower the priority on this for now and remove the milestone.
Maybe the "Bound" status could be delayed, so the RC would be held back for a while and be scheduled only when the PVC is actually ready?
I don't think the RC actually waits for PVC to be bound before scheduling.

I have also this problem using deployment with HELM, I deploy all my pv,pvc,deploy,svc in one Charts, and my pod who requiere a volume from the PVC get scheduled and stay in ContainerCreating mode. I need to wait minutes or delete pods for they correctly get rescheduled using the volume. Here is the log when the pod get automaticly deleted (after minutes) : It would be awesome if the deployment or RC wait for the PVC to be bound before schedule !
may slightly improve this situation as the maximum backoff time of 2 minutes will not fire anymore.
Note that the same error is thrown when node-size == C5 or M5 for kubectl and with kops . Solution is to set node as t2 machines. Due to it seems.
Reading from the new solution is:


Question: allErrs = append(allErrs, field.Invalid(portPath, port.Port, "may not expose port 10250 externally since it is used by kubelet")) } } <ins> if service.Spec.ClusterIP == "None" { allErrs = append(allErrs, field.Invalid(specPath.Child("clusterIP"), service.Spec.ClusterIP, "may not be set to 'None' for LoadBalancer services")) } </ins> case api.ServiceTypeExternalName: if service.Spec.ClusterIP != "" { allErrs = append(allErrs, field.Invalid(specPath.Child("clusterIP"), service.Spec.ClusterIP, "must be empty for ExternalName services"))

Create the following svc: It should fail.
I will try to take a shot at it.
Awesome! Make sure to handle updates in all directions. Historically it is the thing we forget. , Maciej Kwiek wrote:
do you have any examples of this behaviour breaking something? It would help with justifying my PR in code review process :). I know it makes sense to fix this issue, but an example would be nice.
I had a user report thatthey didn't know what was happening - the API should have rejected it with a message that you can't have headless services with load balancers. , Maciej Kwiek wrote:
please what are really use cases where we can use k8s headless service, DB, Proxies .. ?


Question: return fmt.Errorf("logging blocked") } <ins> type cacheSyncWaiter interface { WaitForCacheSync(stopCh <-chan struct{}) map[reflect.Type]bool } </ins> type informerSync struct { <del> sharedInformerFactory informers.SharedInformerFactory </del> <ins> cacheSyncWaiter cacheSyncWaiter </ins> } var _ HealthChecker = &informerSync{} <del> // NewInformerSyncHealthz returns a new HealthChecker that will pass only if all informers in the given sharedInformerFactory sync. func NewInformerSyncHealthz(sharedInformerFactory informers.SharedInformerFactory) HealthChecker { </del> <ins> // NewInformerSyncHealthz returns a new HealthChecker that will pass only if all informers in the given cacheSyncWaiter sync. func NewInformerSyncHealthz(cacheSyncWaiter cacheSyncWaiter) HealthChecker { </ins> return &informerSync{ <del> sharedInformerFactory: sharedInformerFactory, </del> <ins> cacheSyncWaiter: cacheSyncWaiter, </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: for extension API server panics with: Apparently introduces this panic with . What you expected to happen: to do not panic. How to reproduce it (as minimally and precisely as possible): extension API server with - for example the endpoint that the apiserver panics with Anything else we need to know?: Environment: Kubernetes version (use ): / Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind regression
/sig api-machinery
/assign
cc
Oops - sorry for that. It seems it is basically this comment: that I blindly applied without e2e retesting (the problem seem to be lack of tests, which is a bit surprising because bugs in my previous implementations were caught by tests). ^^ BTW - this has been cherrypicked to previous releases: 16: 17: 18: so it's actually even more critical...
I can file the automated cherry-pick PRs once the PR against the master branch is merged. :)
Reopening - since we need to cherrypick it still.
I now opened the cherry-picks: 16 - 17 - 18 -
Fix and picks are merged


Question: * [kubectl](kubectl.md)	 - kubectl controls the Kubernetes cluster manager <del> ###### Auto generated by spf13/cobra at 2015-07-14 00:11:42.956026887 +0000 UTC </del> <ins> ###### Auto generated by spf13/cobra at 2015-08-01 00:12:16.314479678 +0000 UTC </ins> <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->

The current documentation is very vague and doesn't explain what can be updated in this way. Can I rename or stop the pod, change number of replicas in RC etc. For example it seems to be impossible to change pod name:
cc
After an in person discussion with we think the docs should point people to the swagger-ui to find if a field is mutable. I'll add this point into the docs.


Question: } } <del> // applyWaitingForSchedulingGatesCondition adds a {type:PodScheduled, reason:WaitingForGates} condition </del> <ins> // applySchedulingGatedCondition adds a {type:PodScheduled, reason:SchedulingGated} condition </ins> // to a new-created Pod if necessary. <del> func applyWaitingForSchedulingGatesCondition(pod *api.Pod) { </del> <ins> func applySchedulingGatedCondition(pod *api.Pod) { </ins> if !utilfeature.DefaultFeatureGate.Enabled(features.PodSchedulingReadiness) || len(pod.Spec.SchedulingGates) == 0 { return

<img width="844" alt="image" src="" <img width="942" alt="image" src="" Here are some incorrect comments and function name about SchedulingGated Comments and function names need to be changed to SchedulingGated nothing No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scheduling
/remove-kind bug /kind cleanup


Question: if events[i].Type == ContainerChanged { continue } <del> g.eventChannel <- events[i] </del> <ins> select { case g.eventChannel <- events[i]: default: metrics.PLEGDiscardEvents.WithLabelValues().Inc() klog.Error("event channel is full, discard this relist() cycle event") } </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In our stress testing, we found the deadlock issue in PLEG relist() for healthy checking and Kubelet syncLoop() The goroutine stack trace is as following We did some investigation, and found the following issue. The main loop for event processing in Kubelet will check if there are some runtimeErrors: if yes, wait until runtime back to normal; if no, it will call syncLoopIteration to read and process events. And the runtime healthChecks depends on the But in the relist() impl, it will send the pod lifecycle event to eventChannel which may introduce deadlock when eventChannel is full. And in the above code, the eventChannel will not be consumed if the runtime is not healthy. In some situation, the eventChannel may full, and the relist() will hang in sending new pod lifecycle event. And the runtime status will not be able to recovered. It is related to issues What you expected to happen: No such issue How to reproduce it (as minimally and precisely as possible): It is not easy to reproduce, it happens after few days stress testing (scaling the deployment repeatedly with high load). Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): CentOS 7.4 Kernel (e.g. ): Linux bjc-qtt-k8s-node-010 3.10.0-693.2.2.el7.x8664 SMP Tue Sep 12 22:26:13 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig node
Sure, thank you We had some internal fix to break the deadlock, but I am wondering if there are good fix for such kind of issue.
When pleg do relist, can we invoke to judge runtime is ok, if not, not do relist
I submit a PR
Thanks for finding the issue! The issue description makes sense to me. I think PLEG is a core component, we should try our best to unblock it.

The event channel becoming full seems more like an issue in the event consumer side, which shouldn't block PLEG. One solution is to queue the events in PLEG and send later, but that doesn't fundamentally solve the problem, if we don't set a limit on the event number we queue, it seems like a potential memory leak to me; if we set a limit, we may eventually hit this problem again. Another solution is to add a retry for sending event in PLEG, and set a bit in PLEG in between the retry which tells that PLEG is not blocking, it is trying to send out the event. And in the function of PLEG, it returns healthy if the bit is set. Actually, even if we fail to send the event and discard it, since we have already updated the pod cache, the periodical pod resync should eventually pick it up. This may delay pod sync for up to . But if the channel is full, it means that either 1) the syncloop is blocked for some reason, or 2) the node is under heavy loads. In both cases, even the event is successfully sent, the pod sync will likely be delayed anyhow. We can: 1) Enlarge the event channel size to reduce the chance that the channel becoming full. Currently it is arbitrarily set as . 2) Discard the event in PLEG if the channel is already full. In the 3 options above, I'm fine with either 2 or 3. Option 2 doesn't have the 1min pod sync delay, but a bit ad-hoc; option 3 is more generic, but may delay pod sync. There might be other options. /cc Any thoughts?
If PLEG is already blocked by a full event channel, this doesn't help right?
cc/ This looks like the one production issue you mentioned to me a while back.
of options enumerated, option 3 was preferred for me.
Thanks for reporting the issue! what was the load on the node that triggered this condition? If the node wasn't function at all, and the events slowly built up over time, bumping the size of the channel won't help. This should work, and doesn't complicate the logic in PLEG more, so it's my preferred option too.

We need some unit tests, and ideally, a stress test, as mentioned in the sig-node meeting.
Agree, giving a proper size for event channel is tricky as the scale also grows with time. We will ship fix as option 3 () for 1.14. Thanks for summarizing all these up as sig meeting! /assign
If possible, could you share in how large the stress is? It helps understand the issue and kubelet stress handling more. :)
Sure, our product team ( ) will input the stress test data after removing sensitive information in follow up comments.
Thanks for reply. Following is our testcase to reproduce this issue: The test cluster has 5 node on aliyun ECS(like EC2). Three 4C8G machine as master node, Two 96C384G as worker node. Firstly, We create 2 deployment running on the worker node. One deployment is http server pods. Another is siege pods to produce network stress to http server deployment. Those stress test deployments produce 50 CPU Core load on worker node. Then we create a job frequently create/delete 20 pod every 5 seconds. After about 1 day of stess test, The worker node will become NotReady. Dig the pprof goroutine stacktrace found the channel block in .
Yeah, I can see how frequent container state change can cause this issue. The number is very useful, thanks a lot!
cloud you please help to share how to dump the goroutines stack trace of kubelet like what you pasted in your comment. thanks a lot.
This issue is fixed by
using pprof is a answer to my question thank you anyway.


Question: readonly extra_log_files="${LOG_DUMP_EXTRA_FILES:-}" readonly dump_systemd_journal="${LOG_DUMP_SYSTEMD_JOURNAL:-false}" # Log files found in WINDOWS_LOGS_DIR on Windows nodes: <del> readonly windows_node_logfiles="kubelet.log kube-proxy.log docker.log" </del> <ins> readonly windows_node_logfiles="kubelet.log kube-proxy.log docker.log docker_images.log" </ins> # Log files found in other directories on Windows nodes: readonly windows_node_otherfiles="C:WindowsMEMORY.dmp"

When a test run completes it would be useful to have a list of all the test containers that were pulled onto the node - see for the motivation. Let's add a call to at - just after seems like it should work fine. We can dump the output to a file and gather it from each node just like we do for
/assign Yang, we should also consider adding to the log gathering that the GCE diagnostics tool does :)
/sig windows
/reopen in the of our on 03-22 I don't see in the artifacts for the Windows node. Can you check on this? Did we miss a step or is something failing?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Oh, missed to add the new log file to collector list.
Added the fix:


Question: return true } <del> type mockNoopPlugin struct{} </del> <ins> type mockNoopPlugin struct { uuid string } </ins> func (p *mockNoopPlugin) Name() string { <del> return "MockNoop" </del> <ins> return p.uuid </ins> } func NewMockNoopPluginFactory() PluginFactory { <ins> uuid := uuid.New().String() </ins> return func(_ runtime.Object, _ framework.Handle) (framework.Plugin, error) { <del> return &mockNoopPlugin{}, nil </del> <ins> return &mockNoopPlugin{uuid}, nil </ins> } } func TestMerge(t *testing.T) { <ins> m1 := NewMockNoopPluginFactory() m2 := NewMockNoopPluginFactory() </ins> tests := []struct { name            string primaryRegistry Registry

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Below are the failures while running unit test with golang in master branch. Reference : $ go version go version devel go1.17- Thu Apr 22 01:40:02 2021 +0000 linux/ppc64le Note: Occurs in x86 architecture as well. The tests should Pass as with stable go git clone cd kubernetes make test KUBERACE=-race WHAT=./pkg/scheduler/framework/runtime/ Master branch of kubernetes repo. OS: Linux Observed failure on ppc64le and x86 architectures Below is the attached log:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I request help on getting this triaged. Is there a procedure to follow to get this accepted as a bug?
The Go compiler developer has identified the bugs in the test that is causing it to fail. They are listed in the last comment in the Go issue before closing it.
So the correct way to compare function would be something like would you test it in your env?
The issue still exists. Can you please reopen?
I am not sure if you can access this link for our CI job. Can you please check? https://prow.ppc64le- The above tests have been failing every day in our jobs!
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
downloads go from url="${GOLANGVERSION}.linux-" downloads go from "curl -X GET http://build-?project=build_commit -o " I suspect the postsubmit-master-golang-kubernetes-unit-test-ppc64le failed for wrong golang version.

failed for other problems. The TestMerge is not failed. For ppc64le failures, there are several issues in golang: (fixed yesterday)
Yes the go versions for both the jobs are different and it is intentional: : It is meant to run Kubernetes unit tests with the stable version of golang : It is meant to run Kubernetes unit tests with golang built from the master branch of As I mentioned in the description of this issue, these unit test cases are failing only on golang that is built from master branch and passing with stable versions of golang. And as Lynn mentioned in , the golang team has concluded that these failures are due to the bugs in the test case. And the golang issues that you mentioned do not seem to be related to the failures mentioned in this issue.
With , is false Per , do you mean that we should change to ?
No. In general comparing is the right thing to do. But if the Value is actually a , which is not comparable, comparing will panic.
/sig scheduling
Note from golang team about this bug is here:


Question: err = podClient.Delete(context.TODO(), pod.Name, *metav1.NewDeleteOptions(30)) framework.ExpectNoError(err, "failed to delete pod") <del> ginkgo.By("verifying the kubelet observed the termination notice") err = wait.Poll(time.Second*5, time.Second*30, func() (bool, error) { podList, err := e2ekubelet.GetKubeletPods(f.ClientSet, pod.Spec.NodeName) if err != nil { framework.Logf("Unable to retrieve kubelet pods for node %v: %v", pod.Spec.NodeName, err) return false, nil } for _, kubeletPod := range podList.Items { if pod.Name != kubeletPod.Name { continue } if kubeletPod.ObjectMeta.DeletionTimestamp == nil { framework.Logf("deletion has not yet been observed") return false, nil } return true, nil } framework.Logf("no pod exists with the name we were looking for, assuming the termination request was observed and completed") return true, nil }) framework.ExpectNoError(err, "kubelet never observed the termination notice") </del> ginkgo.By("verifying pod deletion was observed") deleted := false var lastPod *v1.Pod

What happened: In reviewing a test for promotion to conformance I noticed the kubelet API was being accessed via the API server's node proxy endpoint (ref: ). This seemed off, so I reviewed . It doesn't explicitly call out kubelet API access as allowed or disallowed. . and agreed kubelet API access seems like a bad idea. I'm now surveying what we have that does_ assume kubelet API access. It's definitely used by parts of e2e framework and some Conformance tests. I'm trying to verify how critical its use is. It appears that much of it is related to debugging or verifying whether kubelets and the control plane agree on who is running what. What you expected to happen: Find a blanket policy that bans kubelet API access for Conformance tests, whether directly, or indirectly through the API server's node proxy endpoint, at any time. Alternatively, find a policy that says kubelet API access may be allowed, but only for debugging or failure purposes (must not be on the critical path to make the test pass) Alternatively, find a policy that says kubelet API access is allowed, but only indirectly through the API server, and evaluated against the same constraints we use for control plane access (GA, non-optional) If we want to say kubelet API access is banned, we can use this as a tracking issue to demote whichever tests rely on it, and come up with a plan to verify the same behavior without kubelet API access. /area conformance /sig architecture /priority important-soon /milestone v1.16 /assign
Look for anything that ends up calling calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test requires kubelet API access to pass may end up calling it if the test fails, via: Some Conformance tests may end up calling this if the test fails specifically Other areas this is used: only if the test fails required for a pre-stop hook test to pass required to make sure the readonly kubelet apis aren't exposed via api server proxy resource usage tracking tests require kubelet api access to pass tests call this in
Looking for doesn't turn up any additional conformance tests, and I don't feel compelled to link the two other tests that use it as they are unlikely to ever be eligible.

The examples pointed out by on slack, the kubelet's endpoints is required to be available for these to pass
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?
are there further changes for this targeted at 1.16?
Any update on this?
/milestone v1.17
Bug Triage Release Team checking in. Code freeze for the 1.17 release is on November 14. Will this make it in? Thanks!
/assign
hi is this still targetting for 1.17 or can we move to 1.18? Code freeze is Nov 14th. thanks!
/milestone v1.18
So we've updated the documented policy. My suggestion to close this out is to drop the [Conformance] tag from the tests that currently require the kubelet API to pass, and open issues that they need to be rewritten. Ignore whether the API is needed for diagnostic info in failure modes. SchedulerPredicates [Serial] calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test Pods should be submitted and removed [NodeConformance] requires kubelet API access to pass
i'd support this ^^
Hello , Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March. Is this issue still intended for milestone 1.18?
Hello There's been no movement on this issue this year. Should this get pushed to the next milestone?
yes, please John T Skarbek <:
/milestone v1.19
/cc


Question: maxImagePullRetries = 5 // Sleep duration between image pull retry attempts. imagePullRetryDelay = time.Second <ins> // Number of parallel count to pull images. maxParallelImagePullCount = 5 </ins> ) // NodeImageWhiteList is a list of images used in node e2e test. These images will be prepulled

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Unless disabled, the tests will prepull all images when running. They do so using the . I think there's the opportunity to improve the runtime of the test suite by prepulling the images in parallel. Why is this needed: Its always nicer to have faster tests :) Specifically for users running locally, it's nice to have the image prepull occur as quick as possible. That being said, we should also be cognizant of any tradeoffs associated with prepulling the images in parallel. If the code is substantially more complex, we would need a large improvement in test speed to justify it. I'm hopeful this discussion could be had once we have a proof of concept diff (which I think should be fairly minimal effort to create).
/sig node
I think this could be a good first issue for someone looking to get involved with contributing to sig-node :) Please post if you plan on working on it, so that there aren't multiple people tackling it.
I'd be interested! I'm looking for a first issue and think this would be a good one
Wonderful! Want to reach out to me on Kubernetes slack? I'm on there too :)
:wave: hiya ! Are you still thinking you could be interested in working on this? Lmk if so, and we can chat about next steps? If not, perhaps I'll open it back up to a newcomer looking for a first commit?
I'm going to open this back up for someone looking for a first commit. Lmk if you're still interest in getting sig-node, and happy to find an issue for you :)
Opening this back up for folks :) I'm happy to mentor whoever picks this up if that'd be helpful :) Ping me on k8s-slack, where I'm also
Can you assign this to meI'd be interested!
/assign
Awesome - feel free to reach out on k8s-slack if you have questions, or if you already have a plan, feel free to tag me in the pull request. Thanks :)
/lifecycle active /priority important-longterm


Question: // `HostIP: s.API.AdvertiseAddrs[0]`, if there is only one address` {Name: "http", ContainerPort: 9898, HostPort: 9898}, }, <ins> SecurityContext: &api.SecurityContext{ SELinuxOptions: &api.SELinuxOptions{ // TODO: This implies our discovery container is not being restricted by // SELinux. This is not optimal and would be nice to adjust in future // so it can read /tmp/secret, but for now this avoids recommending // setenforce 0 system-wide. Type: "unconfined_t", }, }, </ins> }}, Volumes: []api.Volume{{ Name: kubeDiscoverySecretName,

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-beta.11", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-23T22:53:01Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: AWS - CentOS 7.2 x8664 with cloud-init (HVM) (ami-) OS (e.g. from /etc/os-release): NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" Linux ip-172-31-35-103 3.10.0-327.el7.x8664 SMP Thu Nov 19 22:10:57 UTC 2015 x8664 x8664 x86_64 GNU/Linux Install tools: Others: What happened: when selinux is enabled i was getting following error on discovery container What you expected to happen: i expected node to be able to join master, but nothing was running on 9898 port of master i.e.

discovery due the error mentioned above How to reproduce it (as minimally and precisely as possible): use the api which i gave you and do normal kubeadm init Anything else do we need to know: This issue got fixed with setenforce 0
This issue got fixed with setenforce 0 but want to track it
cc
BTW the issue was on latest Centos iso too downloaded from here
cc
this is to do with secrets, and kubelet log is the place to look for more info, could you please grab and paste it in a gist (it's gonna be fairly large)? Also, the kernel audit log would be handy, do you know how to get this (I forgot)?
not sure if this will help
the lines are incomplete, it's best if you redirect it to a file and upload it somewhere with a link here.
could you also provide the output of ?


Question: - mode: 644 - makedirs: true <ins> libltdl7: pkg.installed </ins> docker-upgrade: cmd.run: - name: /opt/kubernetes/helpers/pkg install-no-start {{ docker_pkg_name }} {{ override_docker_ver }} /var/cache/docker-install/{{ override_deb }} - require: - file: /var/cache/docker-install/{{ override_deb }} <ins> - pkg: libltdl7 </ins> {% endif %} # end override_docker_ver != ''

Kubernetes is failing to start on AWS wily. Following the guide from here: Here's a snippet from /var/log/syslog
More logs,
cc
Is there a workaround fix I can use in the scripts to get around this?


Question: bindSource = normalizeWindowsPath(source) } else { if len(options) < 2 { <del> klog.Warningf("azureMount: mount options(%q) command number(%d) less than 2, source:%q, target:%q, skip mounting", </del> <ins> klog.Warningf("mount options(%q) command number(%d) less than 2, source:%q, target:%q, skip mounting", </ins> options, len(options), source, target) return nil } // currently only cifs mount is supported if strings.ToLower(fstype) != "cifs" { <del> return fmt.Errorf("azureMount: only cifs mount is supported now, fstype: %q, mounting source (%q), target (%q), with options (%q)", fstype, source, target, options) </del> <ins> return fmt.Errorf("only cifs mount is supported now, fstype: %q, mounting source (%q), target (%q), with options (%q)", fstype, source, target, options) </ins> } <del> bindSource = source // use PowerShell Environment Variables to store user input string to prevent command line injection // https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment_variables?view=powershell-5.1 cmdLine := fmt.Sprintf(`$PWord = ConvertTo-SecureString -String $Env:smbpassword -AsPlainText -Force` + `;$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $Env:smbuser, $PWord` + `;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Credential $Credential`) cmd := exec.Command("powershell", "/c", cmdLine) cmd.Env = append(os.Environ(), fmt.Sprintf("smbuser=%s", options[0]), fmt.Sprintf("smbpassword=%s", options[1]), fmt.Sprintf("smbremotepath=%s", source)) if output, err := cmd.CombinedOutput(); err != nil { return fmt.Errorf("azureMount: SmbGlobalMapping failed: %v, only SMB mount is supported now, output: %q", err, string(output)) </del> <ins> if output, err := newSMBMapping(options[0], options[1], source); err != nil { if isSMBMappingExist(source) { klog.V(2).Infof("SMB Mapping(%s) already exists, now begin to remove and remount", source) if output, err := removeSMBMapping(source); err != nil { return fmt.Errorf("Remove-SmbGlobalMapping failed: %v, output: %q", err, output) } if output, err := newSMBMapping(options[0], options[1], source); err != nil { return fmt.Errorf("New-SmbGlobalMapping remount failed: %v, output: %q", err, output) } } else { return fmt.Errorf("New-SmbGlobalMapping failed: %v, output: %q", err, output) } </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: azure file remount on Windows in same node would fail What you expected to happen: If user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since command would fail if file share is already mounted on the node: It's related to bug: Error logs would be like following: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9-v1.12, master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign
/sig windows
/sig azure
/milestone v1.14 cc This may be good to backport to 1.12 or 1.13 too if customers are using those versions.