Question: return expectedProbes, expectedRules, err } <ins> probeProtocol, requestPath := parseHealthProbeProtocolAndPath(service) </ins> if servicehelpers.NeedsHealthCheck(service) { podPresencePath, podPresencePort := servicehelpers.GetServiceHealthCheckPathPort(service) <ins> if probeProtocol == "" { probeProtocol = string(network.ProbeProtocolHTTP) } if requestPath == "" { requestPath = podPresencePath } </ins> expectedProbes = append(expectedProbes, network.Probe{ Name: &lbRuleName, ProbePropertiesFormat: &network.ProbePropertiesFormat{ <del> RequestPath:       to.StringPtr(podPresencePath), Protocol:          network.ProbeProtocolHTTP, </del> <ins> RequestPath:       to.StringPtr(requestPath), Protocol:          network.ProbeProtocol(probeProtocol), </ins> Port:              to.Int32Ptr(podPresencePort), IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2),

What would you like to be : Support for additional load balancer health probe configurations (specifically HTTP and HTTPS) for Azure Load Balancers. Why is this needed: The generated Azure Load Balancers for type services only support TCP health probes. Those guarantee that the backend pool consists of nodes listening on the target port but do not validate whether or not anything is responding. We've also noticed elevated volumes SSL handshake errors generated by the TCP health checks targeting 443 when using more verbose log level configurations in ingress-nginx. It would be valuable to configure the type of health probe used by the AKS-created Azure Load Balancer.
/sig azure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any update on this?
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what do you think about this feature requests?
I had a look at a possible implementation. I did not create a PR yet, because the relevant files are in and the said there should only be bug fixes, no new features. In , there is some logic which would lead to the load balancer created with protocol HTTP (which is what we want) The check is from here: and the "RequestsOnlyLocalTraffic" is about the : However, I do not understand the relationship here, why setting should result in HTTP health check. What I think we could do is add a new annotation like and when this is set to , the health check will use the HTTP protocol in .

We could also add an additional annotation to set the property of the health check probe, this could be used so that people can set the path to or similar.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
We have a similar requirement for having an http health probe with a custom path. According to comment, we could get an http probe by setting but our service expects health probes on a different path, not . Looks like proposed solution would work for us too. Are there any workarounds to achieve this currently?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign


Question: size := w.endIndex - w.startIndex var oldest uint64 switch { <del> case size >= w.capacity: // Once the watch event buffer is full, the oldest watch event we can deliver // is the first one in the buffer. oldest = w.cache[w.startIndex%w.capacity].ResourceVersion case w.listResourceVersion > 0: // If the watch event buffer isn't full, the oldest watch event we can deliver // is one greater than the resource version of the last full list. </del> <ins> case w.listResourceVersion > 0 && w.startIndex == 0: // If no event was removed from the buffer since last relist, the oldest watch // event we can deliver is one greater than the resource version of the list. </ins> oldest = w.listResourceVersion + 1 case size > 0: <del> // If we've never completed a list, use the resourceVersion of the oldest event // in the buffer. // This should only happen in unit tests that populate the buffer without // performing list/replace operations. </del> <ins> // If the previous condition is not satisfied: either some event was already // removed from the buffer or we've never completed a list (the latter can // only happen in unit tests that populate the buffer without performing // list/replace operations), the oldest watch event we can deliver is the first // one in the buffer. </ins> oldest = w.cache[w.startIndex%w.capacity].ResourceVersion default: return nil, fmt.Errorf("watch cache isn't correctly initialized")

Opening as an follow up analysis from Merging unexpectedly resulted in a huge flakiness of at least one of the apimachinery tests: That is definitely unexpected, because that PR only changes the size of watchcache, which no e2e tests should rely on. So I started digging deeper into it and indeed that PR also uncovered the bug that was introduced earlier. Here is what is happening: made the watch cache size dynamic however, there was a place in watch cache that relied on the following invariant: "when watchache becomes full, it is full until the next relist" that place is this one: with the change that made watchcache size dynamic, it may no longer be true, i.e.: (a) watch cache was full (b) we are increasing its size (c) it is no longer full, but we no longer can't serve (at least not always) everything since listResourceVersion This is exactly what was happening in the failed tests: the service was updated well back in the past the watchcache initially rotated couple times (no need to increase it) [which pulled our change out of it] the its size was increased that resulted in cache becoming not full so the incoming watch should have returned "too old", but was accepting it (due to the listResourceVersion 0 condition being true), but in fact wasn't able to serve it Bonus point: in "kind" tests we were observing tests taking 30+ minutes - this seem to be a bug in kube-apiserver configuration, because the watch in the test doesn't set TimeoutSeconds, and: (a) in GCE it is default in kube-apiserver to sth random from [5m, 30m]: (b) in kind apparently it isn't... /kind bug /sig apimachinery /assign - FYI
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Opening as an follow up analysis from That is definitely unexpected, because that PR only changes the size of watchcache, which no e2e tests should rely on.

made the watch cache size dynamic however, there was a place in watch cache that relied on the following invariant: the service was updated well back in the past the watchcache initially rotated couple times (no need to increase it) [which pulled our change out of it] the its size was increased that resulted in cache becoming not full so the incoming watch should have returned "too old", but was accepting it (due to the listResourceVersion 0 condition being true), but in fact wasn't able to serve it Bonus point: in "kind" tests we were observing tests taking 30+ minutes - this seem to be a bug in kube-apiserver configuration, because the watch in the test doesn't set TimeoutSeconds, and: (a) in GCE it is default in kube-apiserver to sth random from [5m, 30m]: (b) in kind apparently it isn't... /kind bug /sig apimachinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Working on the fix right now
for now contains a test reproducing the issue Fix coming soon.
is ready for review
it seems TimeoutSeconds is configured explicitely on GCE tests to 300 sec KIND is using Kubeadm under the hood and I could not find any place where those flags are configured, but then I think it should be taken the default values for TimeoutSeconds, no? EDIT Actually, it seems that it does take the default, and min-request-timeout takes precedence, that means it takes 1800s it seems we should set this apiserver option to a lower value, should we do it in KIND itself?
I would want to speak to the kubeadm team first, we avoid deviating from their defaults as much as possible so we can test them.

cc
i see the tests under "client-go should negotiate" are not conformance yet, which makes them less likely to be run outside of k8s testing. i personally don't think our tools used for e2e (GCE/kube-up included) should be configuring these kube-apiserver flags. the tests should behave well with any deployer-X that relies on the apiserver sane defaults.


Question: // cleanupOrphanedPodCgroups removes cgroups that should no longer exist. // it reconciles the cached state of cgroupPods with the specified list of runningPods <del> func (kl *Kubelet) cleanupOrphanedPodCgroups(cgroupPods map[types.UID]cm.CgroupName, runningPods []*kubecontainer.Pod) { </del> <ins> func (kl *Kubelet) cleanupOrphanedPodCgroups(cgroupPods map[types.UID]cm.CgroupName, activePods []*v1.Pod) { </ins> // Add all running pods to the set that we want to preserve podSet := sets.NewString() <del> for _, pod := range runningPods { podSet.Insert(string(pod.ID)) </del> <ins> for _, pod := range activePods { podSet.Insert(string(pod.UID)) </ins> } pcm := kl.containerManager.NewPodContainerManager() // Iterate over all the found pods to verify if they should be running

https://k8s- Only one green run out of the past 11 runs. Potential offenders: /cc (saw you guys discussed this on sig-node slack)
Related test failures: , ,
assigning this to you. and I can help out if necessary.
Still not sure what's going on. In 1 downward api modification failure, I saw a pod created, container started, container killed 1 second later by the kubelet, when it's supposed to run for 120s. Very strange.
See e.g.
So far I've only found the volume tests failing in the ubuntu trusty docker 1.10 vms. Not sure if that's 100% of them or just what I've looked at so far.
One idea is that cgroups are cleaned up before the pod is started, and that somehow causes the pod to fail. See . I'll try to verify my theory later...
That's consistent with my observation. Ubuntu+docker 1.10 seems to be the only one failing. If anyone wants to try reproducing this locally, this would be the image to use.
I noticed that too. The "pod ran to completion" is "pod started and immediately exited". I've tried to reproduce locally and can't find a commit around last Wednesday that obviously would have caused this. I'll post any information I find here.
I tried to reproduce locally (not on GCE) with trusty & docker 1.10 but the tests are all passing so far. Either I didn't wait long enough or it's something specific to the GCE trusty image being used.
-- i am not sure how that is possible. the odd bit to me is we ran w/ pod cgroups on in node e2e since 1.5 if i am not mistaken.
-- ignore my previous comment as previously we only deleted the cgroup when the pod was removed from the api server, and not just terminated. so yeah, its possible something is happening here...
maybe we need to pass in both activePods and runningPods to get a more accurate listing?
see from . Search for pod , which has cgroups cleaned up within 2 seconds of starting. May not be the culplit, but its worth looking into.

the runningPods cache appears like it could have a 2 second latency before updating its cache so it could be related and could be helped by passing in activePods as well...
see:
That is possible. I am also not sure when a pod is considered running, and in what order pods are created. Do we create cgroups before starting the container? Would a pod be returned by GetPods (from above comment) if it had cgroups set up, but the container wasn't running yet? It could also simply be that we are cleaning up a pod's cgroups before it is ever running (if that is possible based on above questions).
-- we create the cgroup before the container is started, so I think we need to pass activePods. see:


Question: opt(klet) } <del> klet.appArmorValidator = apparmor.NewValidator(containerRuntime) klet.softAdmitHandlers.AddPodAdmitHandler(lifecycle.NewAppArmorAdmitHandler(klet.appArmorValidator)) </del> <ins> if sysruntime.GOOS == "linux" { // AppArmor is a Linux kernel security module and it does not support other operating systems. klet.appArmorValidator = apparmor.NewValidator(containerRuntime) klet.softAdmitHandlers.AddPodAdmitHandler(lifecycle.NewAppArmorAdmitHandler(klet.appArmorValidator)) } </ins> klet.softAdmitHandlers.AddPodAdmitHandler(lifecycle.NewNoNewPrivsAdmitHandler(klet.containerRuntime)) klet.softAdmitHandlers.AddPodAdmitHandler(lifecycle.NewProcMountAdmitHandler(klet.containerRuntime))

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If I have the a psp with apparmor annotations: and I deploy a windows pod it gets blocked: What you expected to happen: Apparmor check is ignored for windows pods How to reproduce it (as minimally and precisely as possible): see above Anything else we need to know?: Should be fixed/documented with work around as per Environment: Kubernetes version (use ): kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-06-26T19:51:26Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): windows/unbuntu Kernel (e.g. ): Install tools: aks-engine Network plugin and version (if this is a network-related bug): Others:
/sig windows
/assign I can fix it, it may take some time
I don't actually see anything in that suggests we should ignore windows nodes for apparmor. Can you link to it?
That issue doesn't have guidance explicitly, was more about the need for someone to look into it. I did some investigation with outcomes on the comment and the spread sheet has each PSP detailed with what applies to windows as of now:


Question: return readDockerConfigFileFromBytes(contents) } <ins> // HttpError wraps a non-StatusOK error code as an error. type HttpError struct { StatusCode int Url        string } // Error implements error func (he *HttpError) Error() string { return fmt.Sprintf("http status code: %d while fetching url %s", he.StatusCode, he.Url) } </ins> func ReadUrl(url string, client *http.Client, header *http.Header) (body []byte, err error) { req, err := http.NewRequest("GET", url, nil) if err != nil { <del> glog.Errorf("while creating request to read %s: %v", url, err) </del> return nil, err } if header != nil {

It looks like we always try and hit the metadata server even if we aren't on GCE. We then print an error out. This should either be silent (assume after X attempts that we aren't on GCE?) or should only be activated with an option. Email thread: !topic/google-containers/w-ei5Xs6K0Y
cc:
FYI, the way the credentialprovider logic is set up, providers are asked once per keyring creation whether they are enabled. If they are enabled, then they are asked to provide their contribution to .dockercfg each time the lazy keyring is accessed. For efficiency, there is a caching provider, which composes with heavier providers and stores their .dockercfg contribution for a predetermined TTL. What was happening here was that "Enabled()" for our GCE-metadata implementations was logging about errors fetching "http://metadata", where the entire "Enabled()" check is "err == nil".
I believe that this was fixed by


Question: # Optional: Create autoscaler for cluster's nodes. ENABLE_CLUSTER_AUTOSCALER="${KUBE_ENABLE_CLUSTER_AUTOSCALER:-false}" <del> if [[ "${KUBE_ENABLE_NODE_AUTOSCALER:-false}" == "true" || "${ENABLE_CLUSTER_AUTOSCALER}" == "true" ]]; then ENABLE_CLUSTER_AUTOSCALER="true" </del> <ins> if [[ "${ENABLE_CLUSTER_AUTOSCALER}" == "true" ]]; then </ins> AUTOSCALER_MIN_NODES="${KUBE_AUTOSCALER_MIN_NODES:-}" AUTOSCALER_MAX_NODES="${KUBE_AUTOSCALER_MAX_NODES:-}" AUTOSCALER_ENABLE_SCALE_DOWN="${KUBE_AUTOSCALER_ENABLE_SCALE_DOWN:-false}"

In kube-up we use var while in GKE, documentation, issues etc I think we call it cluster autoscaler. We should make it consistent.
I assume this is what you were already thinking, but: I think we should change it to cluster autoscaler (everywhere). You scale the cluster by adding/removing nodes. You scale a node by adding/removing resources to/from the node, so node autoscaler isn't the right term IMO.


Question: v1 "k8s.io/api/core/v1" discovery "k8s.io/api/discovery/v1beta1" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/tools/record"

Which jobs are flaking: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are flaking: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service Testgrid link: https://k8s- Reason for failure: /sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
CRI-O CI is also hitting this failure. We have been unable to reproduce locally. our CI runs on aws
hmm, I only see 1 failure out of 4 and seems that the connectivity didn't work at all
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: JacobTanenbaum. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Saw the test flaking recently: Wondering if this is just a timeout issue.

Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.

I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.

After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?

3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.

so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.

Would love to help out in reproing this, I'm at a loss for good next step so any suggestions welcome!
that's the problem with these flakes, they are not easy to repro, and with this low rate of failures it may be caused for combination of running tests, environmental problems (cpu, memory, ...) , ... I always start for checking the current failures and try to find patterns
Let's take a look to one occurrence The tests fails because can't connect to the first pod If we check the code of the tests, we can see that it always uses the port as source port, we can use that to identify the nodePort assigned interesting thing here, we are source nating the pod ip, and we don't see any other conntrack entry The test verifies it has endpoints however, kube-proxy is using endpointlsices since this version but it has no entries so the iptables rules to forward the traffic are not there Seems we have to adapt this function in the e2e framework to use endpointslices cc: EDIT seems that this function is being used by all the network tests that can cause flakes if the endpoint is updated before the endpointslice (we switched recently kube-proxy to use the later)
Great catch. Ideally this function should validate if both Endpoint and appropriate EndpointSlice objects(s) exist.
Actually - let me take it and send a PR for it.
is out for review
Interesting failures here It fails to connect to the first pod
Hmm - I took a quick look at kube-proxy logs and these look fine to me....
I clicked too fast, sorry ... I was looking at the wrong service and I though I've found something
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.

just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job:


Question: DefaultKubeletPluginContainersDirName    = "plugin-containers" DefaultKubeletPodResourcesDirName        = "pod-resources" KubeletPluginsDirSELinuxLabel            = "system_u:object_r:container_file_t:s0" <ins> KubeletContainersSharedSELinuxLabel      = "system_u:object_r:container_file_t:s0" </ins> DefaultKubeletCheckpointsDirName         = "checkpoints" )

crio-e2e tests The following tests fail when run on a node with SELinux enabled: Probably since these tests were . No response The host path volumes created under /tmp/hostpathpv don't have the right SELinux label set for access by containers started during the test. No response_ /sig node
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: if s.Items != nil && SchemaHas(s.Items.Schema, pred) { return true } <del> for _, s := range s.Items.JSONSchemas { if SchemaHas(&s, pred) { </del> <ins> for i := range s.Items.JSONSchemas { if SchemaHas(&s.Items.JSONSchemas[i], pred) { </ins> return true } } } <del> for _, s := range s.AllOf { if SchemaHas(&s, pred) { </del> <ins> for i := range s.AllOf { if SchemaHas(&s.AllOf[i], pred) { </ins> return true } } <del> for _, s := range s.AnyOf { if SchemaHas(&s, pred) { </del> <ins> for i := range s.AnyOf { if SchemaHas(&s.AnyOf[i], pred) { </ins> return true } } <del> for _, s := range s.OneOf { if SchemaHas(&s, pred) { </del> <ins> for i := range s.OneOf { if SchemaHas(&s.OneOf[i], pred) { </ins> return true } }

In it has multiple loops that look similar to this: where is of type . By using this range format, it creates a shallow copy of each element in the slice. We can eliminate this allocation by changing to this format: If there is agreement, I'd be happy to open a PR to fix these. Additionally, there are lots of allocations coming from e.g. , . If we are able to change the map type to contain a instead, we could eliminate allocations when ranging through these as well. /kind cleanup /sig api-machinery
/cc
Seems reasonable, jordan and I attacked a number of these a while back on different critical paths, and there was substantial low hanging fruit elsewhere that just wasn’t top of priority. Will be happy to review.
/triage accepted


Question: w.Write(LEVEL_0, "Finalizers:t%vn", pv.ObjectMeta.Finalizers) w.Write(LEVEL_0, "StorageClass:t%sn", storageutil.GetPersistentVolumeClass(pv)) if pv.ObjectMeta.DeletionTimestamp != nil { <del> w.Write(LEVEL_0, "Status:tTerminating (lasts %s)n", translateTimestampUntil(*pv.ObjectMeta.DeletionTimestamp)) </del> <ins> w.Write(LEVEL_0, "Status:tTerminating (lasts %s)n", translateTimestampSince(*pv.ObjectMeta.DeletionTimestamp)) </ins> } else { w.Write(LEVEL_0, "Status:t%vn", pv.Status.Phase) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubectl describe pv xxx shows last<invalidif pv is terminating ! I found pvc and pod have the same issue What you expected to happen: Terminating (lasts 10s) How to reproduce it (as minimally and precisely as possible): 1 create a pv, pvc or pod is terminating 2 using kubectk describe xxx Anything else we need to know?: Environment: Kubernetes version (use ): 1.13.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig cli
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: func main() { command := app.NewSchedulerCommand() <ins> utilflag.InitFlags() </ins> logs.InitLogs() defer logs.FlushLogs()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I pull the latest code, and run with below cmd. But the cluster is not up. And I notice the kube-, What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): GitCommit:"" Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cluster-lifecycle
/assign
Can you please try the fix above?
/assign
/unassign
your pr does not fix it yet? If not, I can help.
The fix from works for me! o/
ok, thanks
reopen , make less modifications.
sorry, i miss hyper...


Question: comm -23  <(go mod edit -json | jq -r '.Replace[] | .Old.Path' | sort)  <(go list -m -json all | jq -r .Path | sort) | <del> xargs -L 1 -I {} echo "-dropreplace={}" | </del> <ins> while read -r X; do echo "-dropreplace=${X}"; done | </ins> xargs -L 100 go mod edit -fmt # prune replace directives that pin to the naturally selected version

I ran update- Not to see a ton of errors Kubernetes version (use ): HEAD of master
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig architecture (code organization / dependencies) /sig testing (hack scripts)
/triage accepted


Question: assert.Nil(t, err, "Expected no error syncing service") // The EndpointSlice marked for deletion should be ignored by the controller, and thus <del> // should not result in more than one action from the client (an update to the non-terminating // EndpointSlice removing the trigger time annotation.) if len(client.Actions()) != numActionsBefore+1 { t.Errorf("Expected 1 more action, got %d", len(client.Actions())-numActionsBefore) </del> <ins> // should not result in any action. if len(client.Actions()) != numActionsBefore { t.Errorf("Expected 0 more actions, got %d", len(client.Actions())-numActionsBefore) </ins> } }

My cluster contains about 4w endpointslices. There are 3 related metrics (endpointslicecontrollerendpointsaddedpersynccount, endpointslicecontrollerendpointsremovedpersynccount, endpointslicecontrollerchanges) to indicates endpoint add remove and update. I found that the apiserver periodically receives lots of endpointslice update requests. In terms of the number of requests, the endpointslice has been fully updated. But from the endpointslice resource itself, it seems that it has not been updated, because "" annotation didn't change It seems that there is no need to update the endpointslice with no really changes. On the other hand, a large number of ep slice update requests will stress the apiserver. In order to make the phenomenon more obvious, the min-resync-period parameter of controller-manager can be adjusted to 1 minute. Create one or more empty endpointslices, and then you can find in the apiserver logs that requests to update these endpointslice every one to two minutes. For clusters with lots of empty endpointslices, this bug may affect the stability of it. No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
there were different improvements in this area, but since 1.20 is going EOL in 8 days it will be good to know if this happens in 1.21 /cc
I'm glad to recieve your comment. I'm not sure because I only have cluster with version 1.20.10. Expect someone who could provide more detail about it. It is actually happening in my cluster, but I don't know how it happens.
I've seen this in 1.20 clusters as well, but have not seen it in 1.21.

My theory was that this is mostly or entirely fixed by Prior to that, any change to EndpointSlice labels would not trigger an EndpointSlice controller sync loop, but the resync period would result in those EndpointSlices being updated at that point. more recent fix () seems like it could also be relevant, but I don't think it matches what's described here. In that case, I believe we're unnecessarily updating EndpointSlices if something irrelevant in the Pod changes (resource version), but I don't believe there's anything special there that would mean we weren't updating EndpointSlices until the resync period. So with all of that said, I believe this is fixed in 1.21+, but let me know if anyone's able to recreate this in those versions and/or if is actually more relevant than it appears on the surface.
Thanks for your reply I will check it.
I confirmed that the periodic updates in my cluster are all empty endpointslices. There are l lot of IPv6 endpointslice in my cluster. But it shouldn't be updated when endpointslice-controller couldn't list pods by service label selector.
/assign
I found that when endpointslice's endpoint array is empty, endpointslice-controller updates it every periodic resync, although it didn't change. I consider it a bug and would like to fork and fix it. The code below could fix the issue of updating empty endpointslices, but I'm not sure if it will introduce new issues. for i := 0; i < len(slicesToDelete); { if len(slicesToCreate) == 0 { break } sliceToDelete := slicesToDelete[i] slice := slicesToCreate[len(slicesToCreate)-1] // Only update EndpointSlices that are owned by this Service and have // the same AddressType. We need to avoid updating EndpointSlices that // are being garbage collected for an old Service with the same name. // The AddressType field is immutable. Since Services also consider // IPFamily immutable, the only case where this should matter will be // the migration from IP to IPv4 and IPv6 AddressTypes, where there's a // chance EndpointSlices with an IP AddressType would otherwise be // updated to IPv4 or IPv6 without this check.

if sliceToDelete.AddressType == slice.AddressType && ownedBy(sliceToDelete, service) { slicesToCreate = slicesToCreate[:len(slicesToCreate)-1] // We must consider the situation that an empty unchanged endpointslice no need to update. If the one to be deleted and the one to be created are the same endpointslice which is empty, skip the update step. // eg: If there are many headless svc with single stack ipv4 for example, a lot of ipv6 empty // endpointslice update could be avoid. if slice.Name != sliceToDelete.Name len(slice.Endpoints) != 0 { slice.Name = sliceToDelete.Name slicesToUpdate = append(slicesToUpdate, slice) } slicesToDelete = append(slicesToDelete[:i], slicesToDelete[i+1:]...) } else { i++ } /cc
Hey I'm not sure I completely understand this fix, but thanks for volunteering to take this on! As I recall, this specific section of the reconciler code is part of the "recycling" logic, where an EndpointSlice marked for creation reuses the name of when marked for the deletion, and the operation becomes an update. I'd appreciate a PR if you can also include a test that recreates the buggy behavior + shows that this fixes it.
Thanks for your reply. I will try more test and show this fix does work.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
I understand the bug now - it looks the root issue is here: Essentially we'd consider the previous placeholder EndpointSlice no longer desired, mark it for deletion, and then that would mean no EndpointSlices would be left, triggering a new placeholder slice to be marked for creation.

The recycling logic you linked above essentially merges creates and deletes into updates, resulting in the behavior you described of empty EndpointSlices always being updated during the resync period. I think this should be relatively straightforward to fix with some changes to the code I referenced above.
/assign
GitHub didn't allow me to assign the following users: spencerhance. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: usingEviction:                   true, getPodFn:                        getPodFn, onDoneFn:                        d.OnPodDeletedOrEvicted, <ins> onFinishFn:                      d.OnPodDeletionOrEvictionFinished, </ins> globalTimeout:                   globalTimeout, skipWaitForDeleteTimeoutSeconds: d.SkipWaitForDeleteTimeoutSeconds, out:                             d.Out,

kubedrain has "onPodDeletedOrEvicted" callback that can be used in custom controllers. It would be very helpful if we add more callbacks that can give more granular control over monitoring, like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed". It would help monitor the draining of nodes on a more granular level. Sometimes while draining, if the pod is stuck during the eviction or deletion of node, there is not much information that custom controllers can get from kubedrain, with the above-mentioned callbacks custom controllers will know which pod is evicting or if it failed during the eviction process.
/sig node
What is the custom controllers in your scenario? Can you elaborate on what you want to achieve with more details. Is this is kubectl improvement, the best sig to involve is sig cli. /sig cli
/remove-sig node please re-apply the sig node if this will end up being a request to sig node
In our case we have in CAPI (). Sometimes while draining, if the pod or volume is stuck during the eviction, there is not much information that we get from kubedrain.Helper. It would be much easier for the controller to report these kinds of issues with more granular callbacks like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed" on kubedrain.Helper.
/assign Assigning this to myself, I have discussed this issue in sig-cli community meeting and they have given green light for this improvement.
/triage accepted /priority backlog


Question: for resource := range hints { if hints[resource] == nil { klog.Infof("[topologymanager] Hint Provider has no preference for NUMA affinity with resource '%s'", resource) <del> allProviderHints = append(allProviderHints, []TopologyHint{{defaultAffinity, true}}) </del> <ins> allProviderHints = append(allProviderHints, []TopologyHint{{nil, true}}) </ins> continue } if len(hints[resource]) == 0 { klog.Infof("[topologymanager] Hint Provider has no possible NUMA affinities for resource '%s'", resource) <del> allProviderHints = append(allProviderHints, []TopologyHint{{defaultAffinity, false}}) </del> <ins> allProviderHints = append(allProviderHints, []TopologyHint{{nil, false}}) </ins> continue }

What happened: I think I'm seeing a problem with the topology manager. I have two numa nodes, 22 cores per node. Kubelet has "--cpu-manager-policy=static --topology-manager-policy=single-numa-node ", and when I try to create a simple pod with 3 cpus and 500Mi of RAM in the guaranteed class it fails. Describing the pod gives "Resources cannot be allocated with Topology Locality", but I know there's lots of CPU and RAM available on both NUMA nodes. I'm instrumenting the topology manager kubelet code, currently there's not enough debug logs to actually be able to tell what's going wrong. What you expected to happen: I expect the pod container to be created on one or the other numa node and get allocated exclusive CPUs (via the static policy) on that NUMA node, and be affined to those CPUs. How to reproduce it (as minimally and precisely as possible): On a kubernetes node with multiple NUMA nodes, enable kubelet with static cpu manager policy and single-numa-node topology manager policy. Try to create a basic pod with a single container in the guaranteed QoS class. Anything else we need to know?: Environment: Kubernetes version (use ): sysadmin chris(keystoneadmin)]$ kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"archive", BuildDate:"2019-10-09T05:41:44Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: wildcat pass hardware, 22 cores per NUMA node, two NUMA nodes. 64GB RAM per NUMA node. OS (e.g: ): CentOS 7 Kernel (e.g.

): 3.10.0-
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
oops, I thought the "close" meant the comment, not the whole issue. :)
I was able to reproduce this issue. The following (new) uint test in fails unexpectedly: With any policy other than the this test passes. This is due to 3 factors: 1) The CPUManager is the only hint provider passing actual valid hints 2) The devicemanager passes back as its hints, resulting in a single (i.e. don’t care) hint 3) We a special casing for the in the device manager that essentially turns the devicemanager's hint into a : Because the single hint from the devicemanager now has a preference, there is no chance to have it combine with any of the CPUManager hints to result in a successful hint. This should ultimately be addressed as part of the following in 1.17 But we should put a one-off patch into the 1.16.x branch to fix this in 1.16.
ptal
I've just updated with my observations. can you have a look and comment ?


Question: defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.VolumeSubpath, false)() for name, test := range cases { <del> _, _, err := makeMounts(&pod, "/pod", &test.container, "fakepodname", "", []string{}, podVolumes, fhu, fsp, nil) </del> <ins> _, _, err := makeMounts(&pod, "/pod", &test.container, "fakepodname", "", []string{}, podVolumes, fhu, fsp, nil, false) </ins> if err != nil && !test.expectError { t.Errorf("test %v failed: %v", name, err) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig windows What happened: When deploying Kubernetes pods, the Linux containers will contain the following entries in the file: # Kubernetes-managed hosts file. 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters 10.240.0.67 dns-querier-1.dns-test-service.e2e-tests-dns- dns-querier-1 However, these are not set on Windows containers; the container's file does not contain any entry. This can be attributed to the fact that individual files cannot be mounted on Windows containers the same way they are on Linux containers. It can be fixed with a change to the kubelet once Windows Server 2019 has shipped, and once CRI-ContainerD is enabled on Windows. We cannot fix it for Windows Server 1803 or earlier with dockershim. What you expected to happen: The file should be managed and populated by Kubelet. How to reproduce it (as minimally and precisely as possible): Spawn a Windows pod, and check its container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cmd /S /C type C:WindowsSystem32driversetchosts Compare the result with a Linux pod's container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cat /etc/hosts Anything else we need to know?: Environment: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: N/A OS (e.g.

from /etc/os-release): Windows Server 1803 (10.0..48) Install tools: acs-engine v0.24.0 Others:
This requires single-file mapping, which won't work in Windows Server version 1803. It should be possible with Windows Server 2019 along with ContainerD
Will it work with Docker on Windows Server 2019?
Not in 18.09 or earlier. I'm not sure what release it will be in because it will require migrating to containerd.
should this issue be tracked for 1.13 in addition to If so please apply the appropriate priority and milestone labels please.
This one needs to be in v1.14. We need to get cri-containerd working first to fix this.
Thanks /milestone v1.14
By default, hosts file are not being used in Windows, it is all registry based. What features is this blocking? CC
The hosts file entries are still taken into account on Windows when it has to resolve a DNS name. This is blocking DNS conformance tests, like.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
Hello! 14 code freeze is coming in about 3 days, is this intended to be implemented in the following week?
No. It is meant to be fixed in the next release, when the Containerd support will be introduced.
/milestone v1.15
Hello! We are starting the code freeze for 1.15 tomorrow EOD. Is this issue still planned for the 1.15 cycle?
This is dependent of the ContainerD implementation. If that hasn't been implemented yet, this can't either.
Is there any update is the containerd implementation finished? Are there any relevant issues or PRs that we can use for tracking?
sounds like a no-go for 1.15, let's just move it out as no one is working on it as well. /milestone v1.16
Hello!

I'm part of the bug triage team for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16? If not please re-tag the issue to the planned milestone.
/milestone v1.17 We won't be able to do this until containerd is ready. It might work in 1.17
Bug triage for 1.17 here. This issue has been open for a significant amount of time and since it is tagged for milestone 1.17, we want to let you know that the 1.17 code freeze is coming in less than one month on Nov. 14th. Will this issue be resolved before then?
We have a PR for it, as you can see above: It;s a pretty small PR too.
The v1.17 code freeze is this Thursday, end of day Pacific time. Will this issue be closed by then?
/milestone v1.18 (code freeze v1.17)
This is going to need a Windows fix, not a ContainerD fix. We won't have that before code freeze on 5 March, so I'm pushing this out to 1.19. /milestone v1.19
This is being tracked with MS internal work item The Windows team thinks an OS fix for Windows Server 2019 will be released with the June patches. I will post updates here as I get them.
Hello from the bug triage team! This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting June 25th (about 4 weeks from now) and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for 1.19, is it still planned for this release?
can you comment on when we can expect a Windows fix for Windows Server 2019 that addresses the single file mount issues in containers when a target file already exists in the container?

we are waiting on a Windows OS fix from Microsoft to proceed here. If that fix comes in time for us to validate this functionality in 1.19 we would like it to be included in 1.19.
Hello! As this issue seems to depend on PR which has been pushed to v1.20, should this issue also be pushed to that milestone?
/milestone 1.20 The OS fix is now expect to be available as part of August Windows patches and then changes to the kubelet will need to follow. /cc
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone.

<detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.20
yes it is expected be part of August Windows release
Hey Bug Triage here. Wanted to follow up on the status of this issue as we're approaching code freeze on 12.11.2020. This issue is tagged for 1.20, is it still planned for this release?
- Yes, we'd like to make it into 1.20. We were waiting on some bugfixes in the Windows OS and those bugfixes are now available (as of Oct 20 2020).
/remove-lifecycle frozen
/priority important-soon
I believe the work involved here is to update to enable this fucntionality for WIndows if container-runtime == remote (still disable this for DockerShim) Update/Add test coverage Update documentation - possible this section ?
/milestone v1.21 Moving to 1.21 since the required OS fixes were not available in time to implement this functionality in v1.20
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hello! I am from the Bug Triage team! This issue has not been updated for a long time, so I'd like to check on the status. The code freeze is starting March 9th, 2021 (about 1 week from now). We want to ensure that each PR has a chance to be merged before code freeze. As the Issue is tagged for 1.21, is it still planned for this release?
Can this be reopened? From a related problem I have experienced using HostAlias for Windows pod deployments, this issue does not seem to be resolved. Instead, it looks as if it was closed by accident because the conversation went cold.
Hello. What Kubernetes version are you running? Also, on the Windows nodes, do you use docker, or containerd?

This is only supported on Windows nodes with containerd, since it supports single file mapping, which is required in order to mount single files into Windows containers.
We have experimented with both 1.22.11 and a 1.23 release and yea, the container runtime is containerd. Any other ideas to check up on?


Question: Containers: []v1.Container{ { Name:  "test", <del> Image: imageutils.GetPauseImageName(), </del> <ins> Image: image, </ins> }, }, },

It's not being used. Alternatively, pass the right image name to start with. /assign /sig scheduling /sig testing
/kind cleanup


Question: v.Visit(s) return s } <ins> // StripNullable returns a copy without nullable. func (s *Structural) StripNullable() *Structural { s = s.DeepCopy() v := Visitor{ Structural: func(s *Structural) bool { changed := s.Nullable s.Nullable = false return changed }, } v.Visit(s) return s } </ins>

With in a CRD we get: This is because gnostic's OpenAPI proto types have no nullable support. Luckily CRDs keep working because in the upper error case we don't stop serving, just set the SSA schema to nil.
Thanks for reporting!
I think it should be "This is because gnostic's OpenAPI v2 proto types have no nullable support.". The problem is that we're not supposed to pass-in v3.
Source:
OK, We'll write a test and fix this, thanks Stefan!
/reopen This still happens in slightly different shape: This comes from :
That's a fairly different bug but yes, thanks!
Is this related to the stating that use of "anyOf" wouldn't be allowed after 1.16? We're seeing issue on OpenFaaS after going to 1.17.5 and 1.18. Usually upgrading K8s versions hasn't caused breaking changes and we're trying to figure out where to start with this one.
No. That post was about structural schemas. v1 CRDs are limited to structural schemas. v1beta1 CRDs can continue to create non-structural schemas until v1beta1 is no longer served (in 1.22) Also, is not completely disallowed, specific restrictions inside an clause (like ) are what are not allowed ("logical constraints using oneOf, allOf, anyOf, not are allowed").
extracted the newly found issue to to track separately with isolated reproducer /close
Closing this issue. <detailsIn response to : extracted the newly found issue to to track separately Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "errors" "fmt" "net" <ins> goruntime "runtime" </ins> "strings" "time" <ins> "github.com/google/cadvisor/machine" "github.com/google/cadvisor/utils/sysfs" </ins> "k8s.io/apimachinery/pkg/watch" "k8s.io/apimachinery/pkg/runtime"

A new flag indicating how kube-proxy gets the number of CPUs, either from goruntime or . In kube-proxy, we can use to set node's , the resulting is . We have nodes where pods with none CPU policy and static CPU policy are running together. Pods like kube-proxy are running in none CPU policy by default, and they only gets CPUs left by static CPU policy pods. If static CPU policy pods take away most CPUs cores, kube-proxy can only get one or two cores, then will be set to a very small values. In this case, a lot packet failure can occur. When we set kube-proxy to work in static CPU policy, kube-proxy only gets the number of CPUs selected by the CPU policy, not the number of total CPUs on the node. This may surprise users who are only trying to limit kube-proxy's CPU usages, but resulting in some conntrack fulls faults. Please let me know if this enhancement is reasonable, so I can send a pr later. Thanks!
/sig network
Is it refer to deploy order of kube-proxy? If kube-proxy deploy first in node, will it get just one or two core?
By one or two I mean kube-proxy gets a small number of CPU cores, because most of CPU cores are allocated to static policy pods, sorry for the misunderstanding. The number of conntrack-max depends on the CPU cores when kube-proxy started, even kube-proxy was first deployed, it can get restarted or redeployed by some reasons. Then the conntrack-max will change.
yeah, when kube-proxy deploy firstly, Maybe it can see the whole cpu nums from shared cpu(/var/lib/kubelet/cpumanagestate when kubelet start with static cpu policy). But I think it is a bug for kube-proxy. The cpu info can also get from node resource info. So will it be suitable to add a flag to set cpu num?
Maybe, or we can add a flag for user to choose where kube-proxy should get CPU informations from. There maybe some multi-tenant scenarios where kube-proxy are deployed for per tenant so they would like to limit conntrack-max by CPU cores allocated to each tenant.
I think the simple fix is to replace with something that counts /proc/cpuinfo. I don't know of anyone running multiple proxies, and I don't think it would work well. The intention was "all the CPUs on the node". PRs welcome!
/triage accepted


Question: if err != nil { t.Errorf("unexpected error: %v", err) } <ins> secretDataNoEmail, err := handleDockercfgContent(username, password, "", server) if err != nil { t.Errorf("unexpected error: %v", err) } </ins> tests := map[string]struct { params    map[string]interface{}

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When I create a docker registry secret, the is also a MUST, I think we should make this parameter as optional as there is no need always put when creating a docker registry secret. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
Agree it should not be required
Did this change get reverted? I do not see as optional on stable.
I feel that this code should be changed because "--docker-email flag" is no longer required.
error: flag docker-email is required. On kubenetes 1.10.3, the above error message still exists
The field is still required. This is very frustrating as does not require an email but Kubernetes does? Is there a good reason for this?
Why is this closed? As far as I can see it is still required?


Question: PromoterE2eRegistry:     "k8s.gcr.io/e2e-test-images", BuildImageRegistry:      "k8s.gcr.io/build-image", InvalidRegistry:         "invalid.com/invalid", <ins> GcEtcdRegistry:          "k8s.gcr.io", </ins> GcRegistry:              "k8s.gcr.io", SigStorageRegistry:      "k8s.gcr.io/sig-storage", GcrReleaseRegistry:      "gcr.io/gke-release",

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: 02/19 20:21 PST Testgrid link: Reason for failure: Image pull failure Anything else we need to know:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows cc


Question: Ports:     []v1.EndpointPort{epp}, }) readyEps++ <del> } else { </del> <ins> } else if shouldPodBeInEndpoints(pod) { </ins> glog.V(5).Infof("Pod is out of service: %v/%v", pod.Namespace, pod.Name) subsets = append(subsets, v1.EndpointSubset{ NotReadyAddresses: []v1.EndpointAddress{epa},

If you create a job in which pods are crashing and a service spawning those pods, this may result in generating extremely huge Endpoints object (which will cause different performance issues). [Based on situation from one of our customers cluster]. After 30m from creating the objects below in my 3 node-cluster I have an endpoint which has almost 1MB of size: Example objects which are enough to cause this : :
- I think that we want to have "failed pods" that have "restart policy" "Always" to be part of "NotReadyAddresses". But do we want to keep pod that are in Failed state and have restart policy "Never"? I think it doesn't make sense. And this would be a quick and easy fix to this particular problem...
Can you update the issue title to "Failed pods with restart policy=Never should not be in the Endpoints.NotReadyAddresses"
done Do you agree that this is a reasonable request?
Geniunely interested: in what case do you want a service and a job? cc:
I can imagine a map-reduce-like thing that serves status from each worker, and uses a headless Service for discovery.
Yes - this is the example I've heard during discussions on last Kubecon.
Well - I don't know if it was map-reduce-like-thing, but it was about headless Service for discovery (and also reporting status from workers).
/assign
GitHub didn't allow me to assign the following users: xiangpengzhao. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We should also not put such pods in the Endpoints.NotReadyAddresses: Succeeded pods with restart policy=Never Succeeded pods with restart policy=OnFailure
- thanks for working on it. I commented on the PR.


Question: } func (nm *NodeManager) refreshNodes() (errList []error) { <del> nm.registeredNodesLock.Lock() defer nm.registeredNodesLock.Unlock() for nodeName := range nm.registeredNodes { </del> <ins> for nodeName := range nm.getNodes() { </ins> nodeInfo, err := nm.getRefreshedNodeInfo(convertToK8sType(nodeName)) if err != nil { errList = append(errList, err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Quote I took goroutine dump of controller-managers that were stuck in volume provisioning and I think we have a deadlock in code: I think we have a problem introduced when we merged - and what is happening is, when in is called it acquires a lock on , but same function ends up calling which acquires another lock on object . Now the problem is, function gets called in a loop and it will try to acquire lock for each node it processes and hence two operations could result in deadlock. This results into inability to provision a with a default StorageClass. The volume does not get to call the API endpoint, just hangs after call, which is clear in the goroutine dump. What you expected to happen: 1) Provision a from successfully. 2) Don't be blocked by credentials secret change. How to reproduce it (as minimally and precisely as possible): 1) Provision a vSphere cluster with credentials secret set 2) Create volume 3) Wait... Anything else we need to know?: Environment: Kubernetes version (use ): v1.19.0-rc.2+-dirty Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux Kernel (e.g. ): GNU/Linux Install tools: openshift-install (kubeadm) Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/vmware
This should be a blocker bug for 1.19 at least as far as vsphere cloudprovider integration is concerned, because it breaks cloudprovider integration/volume provisioning. cc
cc:
/milestone v1.19
Something makes me think this issue existed before, and just escalated to a larger scope after adding informers, demanding refresh for all objects. This is used within plugin in and it is mounting the volume if it is required, calling - it is locking on all the nodes at once. If the pod was just restarted (say just changed, or the health probes demanded a restart), it would process all existing at the same time as trying to mount/create the volume.

And both threads will clang on the same lock, as the would probably not be populated at that moment, so will go the path of - same which is used in . So we get lost volume data, pending etc... And the links I'm sharing are from Kubernetes, where issues with similar footprint randomly arose.


Question: kind: %s metadata: name: %s <del> finalizers: - test-finalizer </del> spec: unknown: uk1 foo: foo1

After running k8s conformance testing on kubernetes v1.27, we have leftover custom resources of the format that should not exist. For example) Basic test instructions: We use the latest official kubernetes kubekins image: We extract kubetest binary from kubekins latest-master docker image. We run serial conformance tests as follows. We also run parallel conformance tests as follows.
/sig api-machinery
The following script is used to cleanup the leftover CRDs:
/triage accepted /assign /cc
/area conformance
/sig architecture


Question: package azure import ( <ins> "strconv" </ins> "testing" "github.com/stretchr/testify/assert"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the specified internal LB uses a subnet with longer name, the LB configuration will fail with following error: What you expected to happen: When constructing the frontend IP configuration name and the load balancing rules name, the subnet name should be truncated if too long. How to reproduce it (as minimally and precisely as possible): Use a subnet that has long name (70 charactors) and create ILB service. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: azure OS (e.g: ): Ubuntu 16.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure /assign


Question: buf := make([]byte, size) buf = buf[:goruntime.Stack(buf, false)] panicReason = strings.TrimSuffix(fmt.Sprintf("%vn%s", panicReason, string(buf)), "n") <ins> // Propagate to parent goroutine panicCh <- panicReason </ins> } <del> // Propagate to parent goroutine panicCh <- panicReason </del> }() if result, err := fn(); err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: has failed to create namespace with error: Related kube-apiserver log says: What I find interesting is "0" status code. What you expected to happen: Nonempty response returned. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): master version, this attempt ran at v1.14.0-alpha.0.793+dc9261bc3b6b81Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): cos Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /cc
/assign
This happened again in:
/cc
- from what I understand, we also know that (at least in some cases) the namespace was actually created. Is that true? [I'm asking, because it significantly reduces the number of places where something could go wrong in those cases]
Yes, at least in the namespace was created despite the 0-length response. All occurrences I saw so far was for creating object of some kind (namespace and replicationcontroller)
It seems we started seeing that quite regularly recently. There was at least one more yesterday (and as in all other cases, it was again on POST operation).
I'm wondering if it may be related to this: In theory, status.Status should only be either StatusSuccess or StatusFailure, but if the data is somewhat corrupted, we would return status.Code = 0.
Though, I went through the code and didn't find anything that could have caused that TBH.
we are seeing this cause many integration test failures, across completely different tests we also saw an odd error that appears to indicate truncated input:
/kind failing-test
Attempting to isolate this with
failed in
- tracking down the earliest time when the "0-length" showed up seems to be at of the CI jobs that shows as failed, the earliest seems to be 2018-11-16 09:24 EST which is a job ( logs are at ). So picking the SHA from that and the one before that ( ) shows the following: seems to include the apiserver panic propagation PR too.

may be you can spot something in this list of changes when you get a chance
prior to the panic propagation PR, if the call in was hit, the kube-apiserver process would exit, so edit on 1/12: root-caused it to this PR. see I'm also not seeing panic output in the logs of those test failures most of the other changes in that range were actually reverted in
ack, if i go back a few ci jobs and pick , i see the timeout thing from sttts -
(creat%7Cupdat%7Cdeploy).*0-length%20response I see this in the kube-apiserver logs from , with no other info/error/panic logs around the request line:
so fake success?
yes, but notably, not a (which was what sttts and I had touched shortly before) also, the time between stageTimestamp and requestReceivedTimestamp is not long enough to have hit the timeout case
i expanded the search range and ended up with this: can you please peek when you get a chance?
the only other thing i can think of is somehow the grpc bump is causing issues
the grpc bump from 1.7.5 to 1.13.0 before the the version of the etcd client we use is still tested with grpc 1.7.5 - are you aware of any grpc changes between 1.7.5 and 1.13.0 that could explain response truncation like this?
In all cases I've seen, the request finished successfully in etcd (i.e. I've seen a couple cases where the request was creation of an object (POST xxx), and the object was then readable from etcd (so it successed). Not sure what etcd returned though..
if grpc is suspected I could add a grpc interceptor whcih would log grpc packets and if the issue is triggered then it would be easier to confirm where truncation occurs in grpc library or higher in the stack. Please let me know if you think it is worth to try.
seen in
that seems to rule out issues on the client-go side as uses directly
agreed
Here's an interesting run from CI about an hour or so ago: Seems to be happening in release-1.13 branch too!

that's not surprising, given the first occurrence was well before code freeze lifted
This seems to be yet another failure mode - most of previous ones were status code 0 - this one has 200.
I believe that also is affected by this. It has failed with and associated kube- is: The kube- entry has 0 status code and the error suggests that response was incomplete.
xref
the logged status of 0 on the server side, and no status stack even with merged indicates that neither Write nor WriteHeader is called. the default go server handles that as a 0-length response with a 200 status code, which is why the server logs a status of 0 and clients see a 0-length 200 response. I tried creating all sorts of errors (metav1.Status errors with and without valid status and code fields, non-status errors, etc), and could not reproduce a case where an error passed to would result in a zero-length response and no log. If the switch statement referenced in was hit with a status that was neither success/failure, and a zero code, the server would attempt to write it and would panic and log in the server log.
with the unit test output fix merged, we're getting better visibility to the occurrences of this:
opened to try to capture more details if any code paths are calling WriteObjectNegotiated and returning without calling Write or WriteHeader
strangely, that PR just , but the check for having called Write/WriteHeader passed and did not log any other details: that might seem to indicate that one of the response writers (we wrap several times, for metrics, auditing, etc) might not be propagating header/body data to the original ResponseWriter, which is super strange
seeing more and more of these failures... for example, 3 out of the last 6 runs of failed on this error
tracked it down and deserves all praise. Fix opened in
and for anyone else in case this comes up again, is at least one underlying cause for the client side 200 when the server responded with a 0. Filed issue to track it on the go-restful side.
Thanks


Question: ReadOnlyPort:      portNumber, PublicAddress:     publicAddress, CacheTimeout:      2 * time.Second, <ins> SyncPodStatus:     true, </ins> }) handler.delegate = m.Handler

Split off as a smaller more specific work item. Apiserver should have a flag that makes it not attempt to GET from kubelets. This would be used in conjunction with . After we have converted the top N Getting-started-guides to the new style, we can rip out the kubelet client from the apiserver altogether.
This is tightly related to so I'll take it.


Question: "encoding/json" "fmt" "net" <ins> "strconv" </ins> "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

Whenever I try to change kube proxy metrics bind using flag , it gives me: Then, I the port , the error changes. I expect the metrics server to listen to 0.0.0.0:. Do I miss something? Environment: Kubernetes version: v1.7.8 Cloud provider or hardware configuration**: on-premise OS (e.g. from /etc/os-release): debian/stretch Kernel (e.g. ): Linux R11U2-7U20N-1 4.9.0-3-amd64 SMP Debian 4.9.30-2+deb9u3 (2017-08-06) x86_64 GNU/Linux Install tools: Others:
/sig network /area kube-proxy
The is being and does NOT allow specifying a port. Have you tried using a config file? The field allows you to specify a port (an example ).
/assign
Turned out we use a wrong type when validating and . Sent for fixing this. Will try to cherrypick the fix as well.
New finding, is broken on HEAD in a different way: Seems to be introduced by
Humm..We have some default logic for : Seems like for some reason it does not auto append the default port number OR the validation happens before defaulting.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
I got the same issue 0.0.0.0 0.0.0.0: both not work with kube-proxy 1.9.3 amd64
It is a BUG. Conflict between componentconfig.IPVar.Set(s string) and validation.validateHostPort.

Could you elaborate the issue a bit more? Which k8s version are you on?
I also meet the same issue with kubernetes 1.8. It's impossibe for Prometheus to fetch metrics from 127.0.0.1: in my case, any workaround? I think we'd better reopen this issue.
metrics binds to by default, are you trying to modify port as well? Thanks for reporting, I confirm the issue is remaining. (or any ip:port) does not work as intended even after the fix. Reopening this and filed a new issue for conflict kubeproxyconfig validation. /reopen
Thanks for reopening the issue. I want to bind metrics to 0.0.0.0: as well. With address 127.0.0.1:, Prometheus cannot pull metrics from this address unless they are on the same node.
is fixing this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale (I just got bitten by this on kube-proxy v1.9.10)
should fix this.
Closing this as is merged. /close
Closing this issue. <detailsIn response to : Closing this as is merged. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/sets" <del> "k8s.io/klog" </del> pluginhelper "k8s.io/kubernetes/pkg/scheduler/framework/plugins/helper" framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" "k8s.io/kubernetes/pkg/scheduler/internal/parallelize"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Logging getting points for ALL priorities set in the default scheduler policy Why is this needed: We have Kubernetes v1.14 with a problem of planning pods for three nodes. The problem is that the scheduler unevenly distributes the pods across the nodes. For example, the CPU on two nodes Node02 and Node03 are 50% loaded, while the CPU on the third Node01 is not loaded at all. While that the pods continue to be planned on Node02 and Node03, but they are not planned on Node01 at all. Scheduler logs (maximum logging level) with uniform planning: and uneven planning: As you can see, the sum of points for the priorities BalancedResourceAllocation, LeastResourceAllocation, SelectorSpreadPriority, TaintTolerationPriority is the same in both cases, but the final result is different. There is no way to understand for what specific policy priorities Node01 did not get 3 points, for to be on a par with the others, because the set of points of some priorities is not logged.
/sig scheduling
I think maybe we missed some logs which can reflect the situation of giving the score. WDYT? If you confirmed, I want to handle this issue.
Thank you for the answer! Here is the full scheduling log (logging level 10): maybe I’m really missing something, but I don’t see information about other priorities here.
Missing scores may be due to some priority functions are using non-map-reduce style, so they didn't hit the printing logic. We can add a logging function there to help you better debugging it. However, the code gets refactored in recent releases, and 1.14 is out of our supporting cycles (only 1.15, ,1.16, 1.17). Can you upgrade to a newer version, or I can provide a code snippet for you to compile on your local 1.14 branch?
Thank you, but I already solved my problem by creating an additional scheduler with a custom scheduling policies for my cronjob. However, it would be cool if you the logging function in the supported versions :)
SG. I created to address this.
I'm running v1.23.8, and I've tried and I'm still not seeing any of the scoring output you in Am I doing something wrong?

Shouldn't this be available in v1.23?
is it possible there is only 1 node candidate after filtering? In that case, the scoring logic will be bypassed as it's meaningless to score for 1 node. If it's not the case, could you check the beginning of the scheduler log, to ensure is configured properly?


Question: hostNetwork: true containers: - name: ip-masq-agent <del> image: gcr.io/google-containers/ip-masq-agent-amd64:v2.0.1 </del> <ins> image: gcr.io/google-containers/ip-masq-agent-amd64:v2.0.2 </ins> resources: limits: cpu: 100m

A number of addon images have CVEs that have been fixed in the upstream base images. We should update these images before the final 1.7 release. cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/cc
cc
The new image appears to have removed . I'm not sure if this was expected, but it caused at least fluentd to stop working on COS (). This might also affect kubeproxy, which uses (a derivative of ).
also the metadata-proxy, prometheus-to-sd, the fluentd-event-exporter, and ip-masq-agent.
anything more to do for this issue?
yes, I need to get merged and update those images again. I also need to bump a few other images (dns and ip-masq-agent).
should be the last update needed.


Question: func (f *fakeTimeBudget) returnUnused(_ time.Duration) {} <ins> func TestStartingResourceVersion(t *testing.T) { backingStorage := &dummyStorage{} cacher, _, err := newTestCacher(backingStorage) if err != nil { t.Fatalf("Couldn't create cacher: %v", err) } defer cacher.Stop() // Wait until cacher is initialized. cacher.ready.wait() // Ensure there is some budget for slowing down processing. // We use the fakeTimeBudget to prevent this test from flaking under // the following conditions: // 1) in total we create 11 events that has to be processed by the watcher // 2) the size of the channels are set to 10 for the watcher // 3) if the test is cpu-starved and the internal goroutine is not picking //    up these events from the channel, after consuming the whole time //    budget (defaulted to 100ms) on waiting, we will simply close the watch, //    which will cause the test failure // Using fakeTimeBudget gives us always a budget to wait and have a test // pick up something from ResultCh in the meantime. // // The same can potentially happen in production, but in that case a watch // can be resumed by the client. This doesn't work in the case of this test, // because we explicitly want to test the behavior that object changes are // happening after the watch was initiated. cacher.dispatchTimeoutBudget = &fakeTimeBudget{} makePod := func(i int) *examplev1.Pod { return &examplev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "foo", Namespace:       "ns", Labels:          map[string]string{"foo": strconv.Itoa(i)}, ResourceVersion: fmt.Sprintf("%d", i), }, } } if err := cacher.watchCache.Add(makePod(1000)); err != nil { t.Errorf("error: %v", err) } // Advance RV by 10. startVersion := uint64(1010) watcher, err := cacher.Watch(context.TODO(), "pods/ns/foo", storage.ListOptions{ResourceVersion: strconv.FormatUint(startVersion, 10), Predicate: storage.Everything}) if err != nil { t.Fatalf("Unexpected error: %v", err) } defer watcher.Stop() for i := 1; i <= 11; i++ { if err := cacher.watchCache.Update(makePod(1000 + i)); err != nil { t.Errorf("error: %v", err) } } select { case e, ok := <-watcher.ResultChan(): if !ok { t.Errorf("unexpectedly closed watch") break } object := e.Object if co, ok := object.(runtime.CacheableObject); ok { object = co.GetObject() } pod := object.(*examplev1.Pod) podRV, err := cacher.versioner.ParseResourceVersion(pod.ResourceVersion) if err != nil { t.Fatalf("unexpected error: %v", err) } // event should have at least rv + 1, since we're starting the watch at rv if podRV <= startVersion { t.Errorf("expected event with resourceVersion of at least %d, got %d", startVersion+1, podRV) } } } </ins> func TestDispatchEventWillNotBeBlockedByTimedOutWatcher(t *testing.T) { backingStorage := &dummyStorage{} cacher, _, err := newTestCacher(backingStorage)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: Reason for failure: Anything else we need to know: /assign
/sig api-machinery /triage accepted
OK - I can quite reliably reproduce this problem by adding before this line: This seems to be a test issue (watch being closed as a result of not being able to process stuff on time). I will think how fix the test and make it not suffer from heavy cpu starvation.
Opened fix in


Question: diskName:         diskName, oldSize:          *resource.NewQuantity(2*(1024*1024*1024), resource.BinarySI), newSize:          *resource.NewQuantity(3*(1024*1024*1024), resource.BinarySI), <del> existedDisk:      compute.Disk{Name: to.StringPtr("disk1"), DiskProperties: &compute.DiskProperties{DiskSizeGB: &diskSizeGB}}, </del> <ins> existedDisk:      compute.Disk{Name: to.StringPtr("disk1"), DiskProperties: &compute.DiskProperties{DiskSizeGB: &diskSizeGB, DiskState: compute.Unattached}}, </ins> expectedQuantity: *resource.NewQuantity(3*(1024*1024*1024), resource.BinarySI), expectedErr:      false, },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: resize Azure disk issue when it's in attached state: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /priority important-soon /sig cloud-provider /area provider/azure /triage accepted /kind bug /assign