Question: "testing" "time" <ins> "github.com/spf13/pflag" </ins> v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime"

introduced configuration of additional logging aspects (e.g. format) through command line flags and config file. This was merged for Kubernetes 1.28. Now the following invocation doesn't work as expected when the config file doesn't configure logging: It should run with verbosity 4. Instead it runs with verbosity 0, from the config. Run with this config file: No response 28 rc n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The problem is caused by this line: At that point, is 4 (from the command line flag), and then it gets overwritten with == 0. The normal approach would be: set up config with defaults modify config through flags modify config further by parsing a config file into it. Can kube-proxy do this? It has the downside that a config file can only overwrite fields if they are in the file. A config field doesn't get reset to defaults just by parsing the file. /cc
it looks like kube-proxy special-cases the to let a flag override config... can/should it do the same thing for verbosity? are there any other flags like this?
I don't think that's the normal approach... it's more normal to make flags and config file mutually exclusive. That's difficult for things which started as flags and moved into config
/sig network /sig instrumentation /kind bug /kind regression /milestone v1.28
let's go with and warn it and comment as much as possible, I was going with something like but Patrick is working on it so I assign to him the issue
/assign
Hey folks, v1.28 Bug Triage Lead here Just wanted to know if this is already tracked by SIG leads and someone already working actively fixing it and set a priority?
Perhaps I should have said "this is how component-base/logs was meant to be used". Somehow it worked for kubelet.

Here's my preliminary solution: It needs more testing, but I wanted to share it early because I am running out of time today.
it is , I can take over Patrick's work in case is needed
It would be nice if the logs API could distinguish between and , how about make to a pointer? But this change is supposed to be a breaking change, I'm not sure of the scope of the impact. For side, if set and at the same time, should the command flag override the config file or warn it and be discarded? I think it should be overridden.
xref:


Question: // TODO The use of 3 states is arbitrary, we could eventually make this test handle "n" states once this test stabilizes. func (scaleTest *HPAScaleTest) run(name string, kind schema.GroupVersionKind, rc *common.ResourceConsumer, f *framework.Framework) { const timeToWait = 15 * time.Minute <del> rc = common.NewDynamicResourceConsumer(name, f.Namespace.Name, kind, int(scaleTest.initPods), int(scaleTest.totalInitialCPUUsage), 0, 0, scaleTest.perPodCPURequest, 200, f.ClientSet, f.InternalClientset, f.ScalesGetter) </del> <ins> rc = common.NewDynamicResourceConsumer(name, f.Namespace.Name, kind, scaleTest.initPods, scaleTest.totalInitialCPUUsage, 0, 0, scaleTest.perPodCPURequest, 200, f.ClientSet, f.InternalClientset, f.ScalesGetter) </ins> defer rc.CleanUp() hpa := common.CreateCPUHorizontalPodAutoscaler(rc, scaleTest.targetCPUUtilizationPercent, scaleTest.minPods, scaleTest.maxPods) defer common.DeleteHorizontalPodAutoscaler(rc, hpa.Name)

Test: https://k8s- Example: https://k8s- The HPA tests for gce-serial are flaking rather regularly; they fail about 2/3 of the time for the last week, and are often the cause of a failed test run. Can we de-flake these? Over multiple test runs the problem seems to be the number of replicas jumping up to 4 unexpectedly: Is the limit not getting set correctly here? Is this an actual bug? /sig autoscaling /priority important-soon /kind failing-test /kind flake
cc:
Recent failures fall into three categories: not starting (, , , , , , ). They fail with error message: increasing unexpectedly (, , ). They fail with error message: waiting for scale down (). This one failed with error message: I suspect 2. and 3. may be caused by test using correct amount of CPU when averaged over minute-long window but spiky (for example using 100% cpu for some time then 0% cpu for rest of the minute). I saw similar behavior when validating recent changes manually. This would result in HPA: behaving as the test expects with before recent changes (because it ignored any spikes during scale*forbidden windows, longer sample windows masking the spike), HPA (correctly) increasing size if spikes align after those changes (failures in group 2.), HPA (correctly) not decreasing size if spikes align after those changes (failure in group 3.), I'll take a look at code of those tests and try to responsible for most of the flakes but I don't have any ideas for a cause yet.
/remove-kind failing-test
I was busy this week but I think I'll be able to prepare a fix for this tomorrow.
/milestone v1.13 /kind failing-test
this tests is failing consistently in
On Friday I verified that CPU usage generated by resource consumer stays really close to the target value but oscillates slightly. To fix this I: Increase CPU usage in the test (resource consumer is implemented in a way that makes me think that deviation fro target are of a fixed size so higher target would make deviation a smaller percent of the target). Lower generated load (the test right now is targeting border between recommendation of 3 instances and 4 instances, I will change the load to something between 2 and 3 instances).

I'm checking if this helps the test.
It seems to have helped. I've sent a PR with those changes for review.
Re-opening until we have at least 3 green runs: https://k8s- /reopen
Reopening this issue. <detailsIn response to : Re-opening because we just had another flake last night: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
But it doesn't work on test grid, the test now fails with: I'll look into this.
Yes, and it's now causing another test to fail as well: https://k8s- Escalating to critical-urgent. This test has been failing for 3 weeks now, and is potentially blocking visibility into other failures. /priority critical-urgent /remove-priority important-soon
This test had a couple of successful runs yesterday but is now flaking/failing in master-blocking: https://k8s- It seems to be passing in master-upgrade. Any chance we could prioritize work on it? It's preventing us from seeing whether fixes for other test failures are working.
There was only 1 flake since merged on Thursday (and ~15 green runs). was an unexpected scale up happening during stabilization. I'll check if lowering CPU utilization a bit lower deflakes the test more but I think it's ok to close this issue now.
Lowering CPU utilization didn't help.
Test jobs where these tests have flaked or failed: https://k8s- (today) https://k8s- (last 3 days ago) https://k8s- (yesterday) https://k8s- (yesterday) https://k8s- (today) https://k8s- (yesterday) ... basically, everywhere the tests run, they're flaky as recently as today. More on Triage: You can see here that across all jobs the HPA tests are flaky, with a "worst flake" perioud of 10/23-10/25, but by no means cleared up.
SIG-Autoscaling's boards are also showing flakiness: https://k8s-, although that looks better than the sig-release test boards. Any idea why?

Thanks for the links I've been looking only at one of those dashboards. I've sent that should reduce flakiness of our tests.
Thanks!
The PR merged. Now let's wait and see if it solves the problem.
The test named in the tile of this issue<sup1</supgot better. Last failures at the dashboards you listed: , (before merged), error before taht was on 10-25), , , . So it looks like the test didn't flake in the last ~2 days. Another two tests<sup2</supare flaking , I'll send a PR that should fix it. I think those flakes are result of behavior of HPA intentionally changing<sup3</supso the PR will just relax expectations of the test. <b1</b[sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability <b2</b[sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale down with External Metric with target value from Stackdriver [Feature:CustomMetricsAutoscaling] and [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale down with External Metric with target average value from Stackdriver [Feature:CustomMetricsAutoscaling] <b3</bHPA no longer delays scale ups based on metrics other than CPU. Flakes are when HPA scales deployment to 2 or 3 so test doesn't observe it having size 1 and times out waiting for that size.
I misread titles of custom metric tests as tests for scaling up somehow. Those flaking tests are tests for scaling down and scale downs sometimes don't happen. I'm looking into why.
While this test has gotten better, it's still flaking some of the time: https://k8s-
I looked at "Latest failures" in the two top categories according to (all the other categories have almost no recent flakes). All of the failures log as if they were not affected by : instead of .

Tests results also say they are running version 1.11... or 1.12... (or more rarely 1.9... or 1.10... but I'm ignoring those for now - they are not related to any changes I made). So it looks like I need to cherry pick the PR into 1.11 and 1.12 branches.
... when I look at git branches it looks like my PR is there. There was with logs that looking like affected it. I'll wait to see if there are more failures in test runs including that change. In case there are I'll add some more logging (I have no ideas for cause of the one failure I saw so far).
/assign
GitHub didn't allow me to assign the following users: jbartosik. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any further update on this? I see it turning into more of a failure than a flake and I see another test failing as well in latest runs
We are due to cut Beta for 1.13 today and would like to understand if this should be a blocker or not.
Also are there cherrypick PRs to backport your change to 1.11 and 1.12 branches?
Potentially related:
6 (consecutive) most recent failures of test do not have judging by their logs. That PR would change them from failures (unexpected scale from 3 to 4 instances during stabilisation) to success (expected 3 to 4 instances during stabilisation). One of the 2 recent failures of looks wrong. One failure was scaling up immediately to 5 instances (wrong). The other was scale up to 4 instances (ok, I'll send a PR relaxing that condition tomorrow). I didn't send PRs cherrypicking test changes to 1.12 (because when I checked it looked like the change was there) not to 1.11 (my changes aren't there so old tests should keep working). merged and it ads more logging, which will help me debug this. Failures in the look like the PR relaxing stability requirement should have helped there too.

the , which has the 6 (consecutive) most recent failures , upgrades the master alone to 1.13(master) and and keep the nodes in old version and then run the old test suite. The latest failing runs show that the tests are run from 1.11.5 branch Doesn't this mean your changes need to be backported to 1.11 branch? That is the reason why you dont see you change in those tests logs
FWIW, the same test also flakes in job which upgrade both master and nodes to 1.13 and runs 1.11 tests.
this failure is currently blocking Beta for 1..13. Can you please investigate these and let us know the path forward.
Fixes done by are all test fixes, not feature fixes. So the problem should be fixed once cherry-pick gets to 1.11. That being said - why are we running 1.11 e2e tests against 1.13 cluster? That assumes full compatibility of both API and implementation. We can never remove a feature without breaking this suite (the tests for the feature will still exist on older branches after all). And the tests are often tweaked to the specifics of implementation. Whenever we make any change to autoscaling algorithm tests need to be updated accordingly. Making tests on 1.11 work well with 1.13 risks they will stop working well with 1.11. Shouldn't the upgrade job create 1.11 cluster, upgrade it to 1.13 and run 1.13 e2e tests on that cluster?
I see these 2 pass in the latest run of https://k8s- Over to to close when he sees fit
We have other flakes on that job now, but HPA is no longer flaking. Thanks so much for the dilligence in getting these flakes cleared up, /close
Closing this issue. <detailsIn response to : We have other flakes on that job now, but HPA is no longer flaking. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: stats, err := hcsshimContainer.Statistics() if err != nil { <ins> if strings.Contains(err.Error(), "0x5") || strings.Contains(err.Error(), "0xc0370105") { // When the container is just created, querying for stats causes access errors because it hasn't started yet // This is transient; skip container for now // // These hcs errors do not have helpers exposed in public package so need to query for the known codes // https://github.com/microsoft/hcsshim/blob/master/internal/hcs/errors.go // PR to expose helpers in hcsshim: https://github.com/microsoft/hcsshim/pull/933 klog.V(4).Infof("Container is not in a state that stats can be accessed '%s': %v. This occurs when the container is created but not started.", containerID, err) return nil, nil } </ins> return nil, err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When reviewing Windows kubelet logs noticed errors like: What you expected to happen: The stats should be returned How to reproduce it (as minimally and precisely as possible): This occurs when the pods has just started or during termination. To simulate the issue: Then deploy pod: Anything else we need to know?: The container is in a state but is has not started running so the call to get the stats fails: This is handled in other places like moby: and Environment: Kubernetes version (use ): 1.18+ Cloud provider or hardware configuration: aks-engine OS (e.g: ): Windows Kernel (e.g. ): 2019 Install tools: Network plugin and version (if this is a network-related bug): Others: Dockershim /sig windows /priority critical-urgent /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc:


Question: cp "${KUBE_ROOT}/cluster/addons/dns/kube-dns/kube-dns.yaml.in" kube-dns.yaml ${SED} -i -e "s/{{ pillar['dns_domain'] }}/${DNS_DOMAIN}/g" kube-dns.yaml ${SED} -i -e "s/{{ pillar['dns_server'] }}/${DNS_SERVER_IP}/g" kube-dns.yaml <ins> ${SED} -i -e "s/{{ pillar['dns_memory_limit'] }}/${DNS_MEMORY_LIMIT}/g" kube-dns.yaml </ins> # TODO update to dns role once we have one. # use kubectl to create kubedns addon ${KUBECTL} --kubeconfig="${CERT_DIR}/admin.kubeconfig" --namespace=kube-system create -f kube-dns.yaml

What happened: The following error occurs when creating kube-dns in local-up- What you expected to happen: hack/local-up- should be able to create kube-dns successfully. How to reproduce it (as minimally and precisely as possible): Run ./hack/local-up- Anything else we need to know?: The memory limit in kube- has been changed to "pillar['dnsmemorylimit']" by But local-up- does not transform it into a value. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing


Question: // AlphaFeatureILBSubsets allows InternalLoadBalancer services to include a subset // of cluster nodes as backends instead of all nodes. AlphaFeatureILBSubsets = "ILBSubsets" <ins> // AlphaFeatureILBCustomSubnet allows InternalLoadBalancer services to specify a // network subnet to allocate ip addresses from. AlphaFeatureILBCustomSubnet = "ILBCustomSubnet" </ins> ) // AlphaFeatureGate contains a mapping of alpha features to whether they are enabled

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: ILB ip addresses come from the subnet that the K8s cluster uses. There is no ability to specify a different subnet. What you expected to happen: ILB ip addresses should be allocated from a different subnet if specified. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: GCE OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area platform/gce /assign
/sig network


Question: }) } <ins> // isInvalidReasonStatusError returns true if this is an API Status error with reason=Invalid. // This is distinct from generic 422 errors we want to fall back to generic error handling. func isInvalidReasonStatusError(err error) bool { if !apierrors.IsInvalid(err) { return false } statusError, isStatusError := err.(*apierrors.StatusError) if !isStatusError { return false } status := statusError.Status() return status.Reason == metav1.StatusReasonInvalid } </ins> // checkErr formats a given error as a string and calls the passed handleErr // func with that string and an kubectl exit code. func checkErr(err error, handleErr func(string, int)) {

kubectl apply displays different errors in v1.21 (and 1.22) vs v1.23 (and above) for the same webhook response, with the error message in v1.23 not being helpful at all. For example, if a webhook returns a status with code is not equal to / the and fields are not populated In v1.21.0, we see the following error. This is a helpful error message with plenty of context. In v1.23.5, for the same webhook status, we'll see: This is not helpful at all. To get an equivalent helpful error message in v1.23.5 Make a webhook return a status with code with or not populated, or not equivalent to . Looking at the code, it looks like this was a consequence of Since this PR was introduced in v1.23, this issue would exist for v1.22 as well. kubectl first checks whether the error message is (in both and ) The definition of was changed in . In 1.21 (and 1.22) just reason "Invalid". Given that the webhook status does not include a , the is executed and kubectl shows a helpful error message. In 1.23 (and above) if OR if the status code is , which is true for the webhook status. Since it’s true, this is executed. is nil, so we just see the response. This error message is not helpful. We should probably update how kubectl handles "invalid" errors and ensure that they display the field too (instead of just restricting to check the ). Verified with v1.21.0 and v1.23.5. Have not verified this for v1.22, but it should also affect v1.22 due to the above reasons. N/A No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign
Hi So this purely a client-side issue, right?

The server is still sending back the exact same response/data structure? Thanks!
FYI, I posted in Kube slack related to this. We were hoping to alter our webhook behavior based on the originating user agent by passing the into the Go context given to our webhook server's request handler. However, it appears we are not able to access the originating user-agent info. As indicated in the Slack thread: In other words, the sooner this issue is fixed, the better.
checked with nikhita offline, will reroute the issue /assign /unassign
fix open in


Question: } for vmssName := range vmssNamesMap { <del> vmss, err := ss.GetScaleSetWithRetry(service, ss.ResourceGroup, vmssName) </del> <ins> vmss, err := ss.getVMSS(vmssName, cacheReadTypeDefault) </ins> // When vmss is being deleted, CreateOrUpdate API would report "the vmss is being deleted" error. // Since it is being deleted, we shouldn't send more CreateOrUpdate requests for it.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Recently we upgraded our cluster from 1.14.4 to 1.14.7. Then we started to observe tons of following errors: I guess the cause may be related to this line which was introduced recently: As we have hundreds of services with LoadBalancer type, and we also enable cluster-autoscaler, the nodes list change very often. For each service, there will be one call to GetScaleSet. The quota will be exhausted quickly. When fails, new nodes created by cluster-autoscaler will lose connectivity to load balancer. What you expected to happen: Kubernetes should not hit VMSS throttling threshold. How to reproduce it (as minimally and precisely as possible): a cluster with aks-engine. Use VMSS. 200 services with LoadBalancer type. up VMSS to 50 nodes down VMSS to 60 nodes scaling up and down VMSS Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.7 Cloud provider or hardware configuration: Azure, VMSS OS (e.g: ): Ubuntu Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure
I create a PR to mitigate the issue.
do we have a rough num about how this PR would mitigate this issue? looks like we need to cherry-pick this PR to old releases. it's better if we have a rough estimation.
The revised version of code calls instead of azure API. The cache refreshes itself every 10 min. I believe this will solve the throttling issue.
I believe we're being affected by this. Apart from getting the - when creating our to use a pre-configured Azure Public IP, the ingress appears to be getting a different IP each time
Thank you all!
The PR is merged. Does your issue remain?

pls cherry-pick that fix to old release, thanks
Cherry picked to release 1.14-1.16
I’ll retry today and let you know Wysłane z iPhone'a
Issue returned in K8S 1.15.4 Warning UpdateLoadBalancerFailed service/nginx-ingress-controller Error updating load balancer with new hosts map[<nodes names]: [EnsureHostInPool(ingress/nginx-ingress-controller): backendPoolID(/subscriptions/<subscription/resourceGroups/<rg/providers/Microsoft.Network/loadBalancers/<rg/backendAddressPools/<rg) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:VMSSGet", EnsureHostInPool(ingress/nginx-ingress-controller):


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/test/e2e/framework"

The test is failing in . https://k8s- https://k8s- https://k8s- The reason is that the test expects that all pause containers () should have . However, in many tests, we are starting test containers with . Those pause containers have . Ideally, containers started in previous test should be cleaned up, so in serial suite it should not affect later test. However, it is not always true when CRI is enabled. For example, in the oom message, we could see: There are 2 problems here: Why are containers not properly cleaned up when CRI enabled? I can't even find which test leaked pause container from the log, because there are many test using pause image, and some of them start a lot of containers, such as density test, restart test. The oom score adj test should not rely on process name . However, I can't find an alternative for now, because we don't report pause container status in our api, we don't even know the container id to inspect with docker or CRI. /cc
This test is still flaking in cri-serial. Some examples: https://k8s- https://k8s- Need to check the logs to see if the symptoms are the same.
It could be that with CRI, it takes kubelet longer to clean up. Or, even worse, that kubelet is not cleaning up properly. We need more information on this on failure. Dumping pods on the node Dumping containers () on the node. Accessing docker directly is not ideal, but it's okay because the test itself only applies to docker. Alternatively, we can get the current running containers from kubelet's /runningPods endpoint. The downside is that this may lag behind the actual state a little bit (seconds). We should also make the test suite run more frequently. It runs once a day, which makes the debugging round-trip overhead too high.
I can reproduce this process leak by running the "Network should recover from ip leaks" on my own node, without CRI enabled. The image used in the test was , and the docker version 1.11.2. The test creates 100 pods, then restarts the docker service by 6 times, with 20s interval in between.

After the test ends, there could be multiple processes running left on the node. Those processes were reparented to pid 1, and showed no containers running. /cc
On the side note, the test should not be blindly checking all processes. It should, instead, take a snapshot of the system before the test and only check the new processes created by the tests.
I filed a new issue for the docker bug since it's not unique to CRI and this test is catching it by accident: Leaving this issue to fix the test by not blindly checking all processes:
The docker bug is not a regression, and is tracked in . The problem of the test is not necessarily to be fixed in v1.6, at least it's not release blocker. :)


Question: return res } <ins> func newEphemeralStorageResourceList(ephemeral, cpu, memory string) v1.ResourceList { res := v1.ResourceList{} if ephemeral != "" { res[v1.ResourceEphemeralStorage] = resource.MustParse(ephemeral) } if cpu != "" { res[v1.ResourceCPU] = resource.MustParse(cpu) } if memory != "" { res[v1.ResourceMemory] = resource.MustParse("1Mi") } return res } </ins> func newResourceRequirements(requests, limits v1.ResourceList) v1.ResourceRequirements { res := v1.ResourceRequirements{} res.Requests = requests

It is possible to have multiple signals that point to the same type of resource, e.g., both SignalNodeFsAvailable and SignalAllocatableNodeFsAvailable refer to the same resource NodeFs. Change the map from map[v1.ResourceName]evictionapi.Signal to map[v1.ResourceName][]evictionapi.Signal
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/kind cleanup /priority important/soon taking a guess at labels to satisfy the milestone munger
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Looks like the PR for this is just awaiting merge


Question: &Foo{}, &FooList{}, ) <ins> metav1.AddToGroupVersion(scheme, SchemeGroupVersion) </ins> return nil }

Is this a BUG REPORT or FEATURE REQUEST?: Bug report /kind bug What happened: I'm following the directions to run the sample controller on When running it as directed, it gives these errors: E1214 10:34:25. ] Failed to list v1alpha1.Foo: v1.ListOptions is not suitable for converting to "" E1214 10:34:26. ] Failed to list v1alpha1.Foo: v1.ListOptions is not suitable for converting to "" What you expected to happen: Not to generate these errors. How to reproduce it (as minimally and precisely as possible): 1) Have a working kube config. 2) checkout kubernetes/sample-controller 3) run it: Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-09-14T06:55:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.0.845+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-10-02T22:37:04Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): my workstation is 14.04.5 LTS, Trusty Tahr Kernel (e.g. ): my workstation kernel is 4.4.0-96-generic Install tools: Others: go version 1.9
cc - Hi, it looks like you checked this in with . What all do I need to do to get this to work?
/sig api-machinery
Hey - thanks for this report. I'm not sure how, but we were missing an extra line in I've put in to resolve this (and it's working for me now :tada:)


Question: m.incoming <- Event{action, obj} } <ins> // Action distributes the given event among all watchers, or drops it on the floor // if too many incoming actions are queued up.  Returns true if the action was sent, // false if dropped. func (m *Broadcaster) ActionOrDrop(action EventType, obj runtime.Object) bool { select { case m.incoming <- Event{action, obj}: return true default: return false } } </ins> // Shutdown disconnects all watchers (but any queued events will still be distributed). // You must not call Action or Watch* after calling Shutdown. This call blocks // until all events have been distributed through the outbound channels. Note

The standard event recorder, when recording an event, calls the underlying Broadcaster in a goroutine, so that even recording is non-blocking. Things like leader election emit events over the course of their lifecycle via this broadcaster. Consider the following scenario: GR1: spawn leader election ----GR2 GR2: Leader election: emit event "Lease Acquired" ----spawn GR3 () GR1: Shutdown leader election ----blocks until GR2 is done GR1: Shutdown broadcaster --() GR3: ----send on closed channel () Practically, we've observed this behavior popping up occaisionally in tests in controller-runtime:
/sig api-machinery /kind bug
<details<summaryrelevant race detector output/stack traces</summary</details
were you planning on sending a fix, or need our help to send one to fix it?
I can probably try to figure out a fix if you want, but the whole model for how event broadcasters currently work is inverse to how Go suggests you manage channels (only writers should ever close channels, not readers). I'll need to do a bit of playing around with things.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think this is fixed by . Please reopen if it's not the case.
/triage accepted
Would be possible to cherry-pick this fix in 1.20?


Question: deletedDir := fmt.Sprintf("%s040(deleted)", dir) return ((mp.Path == dir) || (mp.Path == deletedDir)) } <ins> // PathExists returns true if the specified path exists. // TODO: clean this up to use pkg/util/file/FileExists func PathExists(path string) (bool, error) { _, err := os.Stat(path) if err == nil { return true, nil } else if errors.Is(err, fs.ErrNotExist) { err = syscall.Access(path, syscall.F_OK) if err == nil { // The access syscall says the file exists, the stat syscall says it // doesn't. This was observed on CIFS when the path was removed at // the server somehow. POSIX calls this a stale file handle, let's fake // that error and treat the path as existing but corrupted. klog.Warningf("Potential stale file handle detected: %s", path) return true, syscall.ESTALE } return false, nil } else if IsCorruptedMnt(err) { return true, err } return false, err } </ins>

What happened: I was running the subpath testsuite against Azurefile. This test failed and can be easily reproduced: Basically what this test does is: a PVC use azurefile a pod that has two containers. One of it mounts the volume itself. The second container mounts the subpath of the volume. the subpath dir from the container that has the whole volume the pod, the pod will stuck in terminating status and cannot be cleaned up other than What you expected to happen: The pod can be deleted successfully. How to reproduce it (as minimally and precisely as possible): the pvc the pod the subpath dir from the parent pod the pod Anything else we need to know?: The volume is being formatted as cifs type Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: Azurefile-csi-driver 0.9.0 OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux nodepool-- 5.4.0-1016-azure -Ubuntu SMP Thu Jun 4 22:49:15 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /cc /cc
log After debugging with and offline, the issue is that: After the dir from the subpath was removed, the subpath mountpoint on the host machine becomes into a weird state where you can still find the dir by doing a readdir to its parent. But doing or something like that will result in a "No such file or directory". When you do a ls on the path:
/triage accepted
So the issue from code is that It will fail the path exists check while the actual mountpoint exists. So when the code logic runs to here: It will fail to remove the container dir because it somehow detect there is still file there. To fix this as a workaround, user can manually , and then the kubelet can take care of this. To fix it ultimately, we need to see if this is an issue specific to cifs filesystem? Or we should consider do an umount regardless if the path exists.
/assign
/assign
Tried the same scenario using NFS. It works as expected using NFS.
Yeah, that's why I am wondering if this is something special with CIFS?

Looks like it has something to do with the azurefile-csi storage class mount options. I was getting the below error when I was trying to do ls on the subpath. So I realized there might be some issue with the file permissions. root ls ls: cannot open directory '.': No such file or directory When I removed the dirmode and filemode properties from the storageclass and performed the same steps as mentioned in the issue, the pod got deleted successfully.(Although it took a lot of time 10-15 mins for the pod to get deleted)
FYI, I am planning to use this PR to fix the unmount issue in Azure File & SMB CSI driver: Not sure how to handle gracefully in k8s subPath code since that's common code.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
So this PR does not fix the Azurefile subpath test issue?
no, that PR only fixed the unmount issue inside CSI driver. While for the subPath issue, we should fix it in k8s upstream code as you mentioned above:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
back to original discussion, shall we use instead of inside func? That will solve the issue of smb file PathExists check issue.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this subPath issue was fixed in , ,


Question: } func LoadAppArmorProfiles(f *framework.Framework) { <del> _, err := createAppArmorProfileCM(f) framework.ExpectNoError(err) _, err = createAppArmorProfileLoader(f) framework.ExpectNoError(err) </del> <ins> createAppArmorProfileCM(f) createAppArmorProfileLoader(f) </ins> } // CreateAppArmorTestPod creates a pod that tests apparmor profile enforcement. The pod exits with

The "AppArmor load AppArmor profiles should enforce an AppArmor profile" e2e test in failing on our 5k-node gce clusters - https://k8s- (run no. 8) due to timeout: Seems like the test is trying to load the profiles across all the nodes using a daemonset and then running a test pod to check that. It's passing on our 2k-node clusters (https://k8s-) but failing for 5k - suggesting that it's likely due to size of the cluster. I think it's the ds creation taking longer on large cluster. cc
Seems like you're the most relevant here. Could one of you take a look into it? And does increasing the timeout for the test-pod based on no. of nodes sound reasonable?
Assigning it you since is OOO. Feel free to reassign as apt.
This is part of the correctness suite and large cluster tests are release-blocking.
FYI
Reassigning to who is back on Tuesday. He wrote the AppArmor stuff and should be able to debug this quickly. /assign
Ref
Could you manage to take a look into this?
Sorry, I meant to reply to this! I have a fix, just need to confirm it works. I think you're right about the DaemonSet timing out. There's no reason this needs to load the profiles on every node, so the fix just runs a single loader, and uses pod affinity to co-schedule the test pod.
Smooth!
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: go_library( name = "go_default_library", srcs = [ <ins> "conversion.go", </ins> "deepcopy.go", "doc.go", "generated.pb.go",

Making an API request for table output and specifying the parameter causes a server error in parameter conversion, and an inaccurate error message in response: server log: E0804 09:41:14. ] apiserver received an error that is not an metav1.Status: converting (url.Values).[]string to (v1alpha1.TableOptions).v1alpha1.IncludeObjectPolicy: couldn't copy '[]string' into 'v1alpha1.IncludeObjectPolicy'; didn't understand types
Ugh, typed strings.


Question: pkg_tar( name = "kubernetes-cni-data", <del> package_dir = "/opt/cni", </del> <ins> package_dir = "/opt/cni/bin", </ins> deps = ["@kubernetes_cni//file"], )

Is this a BUG REPORT or FEATURE REQUEST?: What happened: Kubeadm E2E test is currently broken due to the commit I report the same issue on my local cluster. If I revert PR , everything works fine. We believe the current weave-net plugin doesn't work with the CNI version change. Right now all kubeadm related development is blocked on this. Could we consider reverting this change from head until we figure out what's wrong in weave-net/CNI? More context in
/assign
Please reopen this in kubernetes/kubeadm. There is an issue as e2e tests & PR job are failing, but locally the CNI bit seem to work for me, so I think it has something with building the debs to do. Not technically. We just don't have any coverage right now. Not IMO. We just need to forward-fix the issue. Closing this; you can reopen in kubernetes/kubeadm
kubeadm e2e test still red. Do you have any progress or hint in terms how to fix it forward? Just to clarify, If I install k8s and kubeadm locally using deb, I can repro the issue on my test box. I could believe it's sth wrong with debs. In the meantime, CNI and plug-in like weave-net change is a big deal. We clearly didn't test it thoroughly for this PR. Reverting the change seems to be the clean solution.
FYI kubernetes/release bumps new CNI releases for deb/rpm packages.
Thanks.
Is this the fix needed to fix these tests?
I actually think that will take us in the other direction. During the kubeadm e2e tests, we use the deb packages built by bazel. During our official release process, we use the debs built by anago (in the release repo). Bumping CNI in our bazel build broken our e2e tests. Bumping CNI in the release repo scripts will similarly break our official release builds as well. If I understand the problem correctly, the only way to fix both of them is for the CNI providers like weave-net to be updated to handle the changes of upgrading CNI to 0.6.0.
I think this issue worth some more discussion.

Re-open the case as it might not be affecting kubeadm only
sorry, i cannot reopen the issue as it's closed by But let's continue to discuss the option here.
Yes, I think so
This will be fixed by ; this was due to the CNI binaries being extracted in the wrong place. It is kind of a kubeadm packaging issue. (Reopening just to make the PR close this)
nice find!


Question: name                      string rejectForwardingRedirects bool serverStatusCode          int <ins> redirect                  string </ins> expectStatusCode          int expectBody                []byte }{

Upgraded from Kubernetes v1.25.0 to v1.25.1. Standard metrics-server deployment now causes the following message to appear in the kube-apiserver logs on a recurring basis: Adding a quick hack debug Infof print shows: It appears that the apiserver is sending an If-None-Match cache control header when refreshing the openapi spec, and the metrics-server service is correctly responding with a 304 Not Modified. No errors logged; proper refresh of OpenAPI specs Deploy metrics-server, or any other service that uses apiserver aggregation and honors caching headers such as , to a Kubernetes 1.25.1 cluster. made bad assumptions; not all HTTP 3XX responses are redirects. It should probably only reject the response if it is 300-399 AND the header is set. <details</detailsnone <details</detailsk3s containerd flannel
Taking a stab at some tags here... /cc /cc /cc /area apiserver /sig api-machinery
thanks for the report, looks like this should be /priority critical-urgent /kind regression cc since this requires a respin of a final 1.22 patch release :-/
Fix open in , will get cherry-picks open to start greening up CI
/triage accepted
I need to make a redirect to the external URL but it's not allowed :(


Question: err := f.FederationClientset.Services(nsName).Delete(service.Name, &metav1.DeleteOptions{}) framework.ExpectNoError(err, "Error deleting service %q in namespace %q", service.Name, nsName) }() <ins> By(fmt.Sprintf("Wait for service shards to be created in all clusters for service "%s/%s"", nsName, service.Name)) </ins> waitForServiceShardsOrFail(nsName, service, clusters) <ins> framework.Logf("Successfully created and synced service "%s/%s" to all clusters", nsName, service.Name) By(fmt.Sprintf("Update federated service "%s/%s"", nsName, service.Name)) service = updateServiceOrFail(f.FederationClientset, nsName, FederatedServiceName) waitForServiceShardsOrFail(nsName, service, clusters) framework.Logf("Successfully updated and synced service "%s/%s" to clusters", nsName, service.Name) </ins> }) It("should be deleted from underlying clusters when OrphanDependents is false", func() {

As mentioned in when the service is deleted from the underlying cluster, the controller should recreate it, but it does not. Below's a transcript. is talking to the federation-apiserver, and is talking to the normal cluster it's pointed at.
Thanks for catching this cc
cc
This is definitely a bug, but I think that we can kick it out into v1.4. Until fixed (i.e. while in beta), we can tell customers not to delete the underlying services in their clusters.
it should not be, I have tested this before. How long did you wait when you check the service existence in last step? Can you send me the logs? The service deletion will cause federation service update, and then the federation service watcher will catch the update even and call another round of service updates to all clusters. When you delete the service, if the service got ingress ip already? I am thinking if the deletion did not trigger federation service update? So so no update event generated..
I think that the best way forward will be to encode this in a reproducible e2e test, otherwise there are too many variables to debug effectively. It's great to know that you've tested this case, and it works in at least some scenarios.
one possible solution come into my mind, currently we set the service full sync time to 10 minutes , if we change it back to 30s, then at least during the full sync, the deleted service will be re-created. I will send a PR anyway, you can judge if it should be put to v1.3
I think we need to find and fix the underlying problem, not cover it up with a shorter sync period. If we have the correct watch on each service, we will be notified as soon as they are deleted. Why is the watch notification not firing correctly?
Keeping this open until we've verified this fix in e2e tests.
I believe this to be fixed, but we still lack an e2e test to verify conclusively. would you like to take this on along with the other e2e tests that you are busy adding?
ping
plan on doing this for 1.4? or bump to 1.5?
Yes, 1.5 is fine.
switching this to 1.6 as it's too late for 1.5. ok?

Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.


Question: } // wait for deployment to create some rs ginkgo.By("Wait for the Deployment to create new ReplicaSet") <ins> var replicaset appsv1.ReplicaSet </ins> err = wait.PollImmediate(500*time.Millisecond, 1*time.Minute, func() (bool, error) { rsList, err := rsClient.List(context.TODO(), metav1.ListOptions{}) if err != nil { return false, fmt.Errorf("failed to list rs: %v", err) } <del> return len(rsList.Items) > 0, nil </del> <ins> if len(rsList.Items) > 0 { replicaset = rsList.Items[0] return true, nil } return false, nil </ins> }) if err != nil { framework.Failf("Failed to wait for the Deployment to create some ReplicaSet: %v", err) } <ins> desiredGeneration := replicaset.Generation if err := wait.PollImmediate(100*time.Millisecond, 60*time.Second, func() (bool, error) { newRS, err := clientSet.AppsV1().ReplicaSets(replicaset.Namespace).Get(context.TODO(), replicaset.Name, metav1.GetOptions{}) if err != nil { return false, err } return newRS.Status.ObservedGeneration >= desiredGeneration && newRS.Status.Replicas == *replicaset.Spec.Replicas, nil }); err != nil { framework.Failf("failed to verify .Status.Replicas is equal to .Spec.Replicas for replicaset %q: %v", replicaset.Name, err) } </ins> ginkgo.By("delete the deployment") deleteOptions := getOrphanOptions() deleteOptions.Preconditions = metav1.NewUIDPreconditions(string(createdDeployment.UID))

Hi, first I have to mention I didn't saw this in Kubernetes CI, we're running the conformance tests on our internal platform and from time to time the test fails. Which test(s) are flaking: Reason for failure: The test works like this: create deployment wait until replicaset is created delete deployment (with orphan) count if we have 0 deployments, 1 replicaset and 2 pods In our environment it fails because from time to time we don't already have 2 pods. Imho this happens because the test doesn't wait until all Pods are created before deleting the Deployment Proposed
/sig apps These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/sig api-machinery (as per the test location)
/cc /sig apps
/milestone v1.19 Since I milestoned


Question: cpuPolicyName              string nodeAllocatableReservation v1.ResourceList isTopologyBroken           bool <del> panicMsg                   string </del> expectedPolicy             string expectedError              error skipIfPermissionsError     bool

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When enable cpumanager by , but not set and , kubelet will get panic error in log What you expected to happen: Should graceful exit with error not panic. How to reproduce it (as minimally and precisely as possible): set and but not set and Anything else we need to know?:
/sig node
Great that it's not panicing anymore, but I'd also love to get the "requires + to be greater than zero" hint in this particular case. It looks like even with this doesn't seem to be the case (since it errors out with the non-actionable "unable to determine reserved CPU resources for static policy". Should I open another issue for this?


Question: // setPodHostAndAnnotations sets the given pod's host to 'machine' if and only if it was // previously 'oldMachine' and merges the provided annotations with those of the pod. // Returns the current state of the pod, or an error. <del> func (r *BindingREST) setPodHostAndAnnotations(ctx context.Context, podID, oldMachine, machine string, annotations map[string]string, dryRun bool) (finalPod *api.Pod, err error) { </del> <ins> func (r *BindingREST) setPodHostAndAnnotations(ctx context.Context, podUID types.UID, podResourceVersion, podID, oldMachine, machine string, annotations map[string]string, dryRun bool) (finalPod *api.Pod, err error) { </ins> podKey, err := r.store.KeyFunc(ctx, podID) if err != nil { return nil, err } <del> err = r.store.Storage.GuaranteedUpdate(ctx, podKey, &api.Pod{}, false, nil, storage.SimpleUpdate(func(obj runtime.Object) (runtime.Object, error) { </del> <ins> var preconditions *storage.Preconditions if podUID != "" || podResourceVersion != "" { preconditions = &storage.Preconditions{} if podUID != "" { preconditions.UID = &podUID } if podResourceVersion != "" { preconditions.ResourceVersion = &podResourceVersion } } err = r.store.Storage.GuaranteedUpdate(ctx, podKey, &api.Pod{}, false, preconditions, storage.SimpleUpdate(func(obj runtime.Object) (runtime.Object, error) { </ins> pod, ok := obj.(*api.Pod) if !ok { return nil, fmt.Errorf("unexpected object: %#v", obj)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The api bind a pod to a node. But the pod is specified by name (not uid), which is not a unique identifier for pod. So the caller of can't make sure the pod is exactly what he want. api can bind a pod with specified uid to a node. If the uid not match, return an error a pending pod by statefulset named busybox, make sure it's pending, the scheduler will not bind it(use non-existed node selector or other method). So we will get a pending pod busybox-0. following code the in the code, delete the pod busybox-0 manually during the sleep or other method. The statefulset busybox will create another pod busybox-0 with different uid , the api in the code above will bind an unexpected pod to node The logic in scheduler is the same. If the pod is recreate with the same name and different resource require or selector between scheduler's and , scheduler will do a wrong bind for the pod. I think the root cause is the api does not check the uid. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.0-alpha.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-20T09:01:41Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.9-aliyunedge.1", GitCommit:"", GitTreeState:"", BuildDate:"2021-03-08T09:56:01Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/sig scheduling for triage (the binding feature of (pod to node) is for scheduling).
For the sake of precedence, are there other APIs that are based on object UID?
uids are used as a precondition is used in many places. if the server started checking that a uid on the binding object (if specified) matched the uid of the pod being modified, that seems sufficient
is there any way for the caller of Bind to confirm that he is binding the right pod? or not uid, use resource version?
that would require: the server to check that bind UID (if specified) matches the pod UID (passing the binding uid from BindingREST#Create → assignPod → setPodHostAndAnnotations and verifying the uid match in setPodHostAndAnnotations) the binding API caller to set in the posted binding object (as illustrated in the description of this issue)
actually, kube-scheduler(the most common binding API caller) has set the in the posted binding object. but the server doesn't check the uid. in my program, i do those to work around(not elegant, but it works...): call the binding API with an uid in annotation(i found that BindingREST#Create → assignPod → setPodHostAndAnnotations will carry annotation). check the matchs , if not match, delete the pod and wait for an new pod ... from your point of view, is it necessary to ensure the pod is the caller want when someone call bind API? what's your advice for this issue
given the scheduler made decisions based on the current shape of the pod spec, I think it makes sense for it to work to make sure it is binding that instance of the pod I would advocate for supporting uid as a precondition in the binding API and starting to use it from the scheduler
i agree with you
/remove-sig api-machinery
I also agree with this and this cannot achieve only on sig/scheduling. Sorry, if it is not suitable, but I'd like to add sig/api-machinery label again.

/sig api-machinery
Either scheduling or node sigs own the pod binding API. Either of those would be a good place to coordinate adding a fix to honor uid as a precondition API machinery owns the generic API plumbing and API server behavior, not implementations of most specific APIs like the pod or pod binding APIs /remove-sig api-machinery
Given you already exercised Jodan's suggestion locally, would you mind raising an offical PR?
how about this PR


Question: } } <ins> func TestInvalidSelectors(t *testing.T) { testCases := map[string]struct { labelSelector *metav1.LabelSelector }{ "illegal value key": { labelSelector: &metav1.LabelSelector{ MatchLabels: map[string]string{ "k8s.io/too/many/slashes": "value", }, }, }, "illegal operator": { labelSelector: &metav1.LabelSelector{ MatchExpressions: []metav1.LabelSelectorRequirement{ { Key:      "foo", Operator: metav1.LabelSelectorOperator("illegal"), Values:   []string{"bar"}, }, }, }, }, } for tn, tc := range testCases { t.Run(tn, func(t *testing.T) { dc, ps := newFakeDisruptionController() pdb, pdbName := newMinAvailablePodDisruptionBudget(t, intstr.FromInt(3)) pdb.Spec.Selector = tc.labelSelector add(t, dc.pdbStore, pdb) dc.sync(pdbName) ps.VerifyPdbStatus(t, pdbName, 0, 0, 0, 0, map[string]metav1.Time{}) }) } } </ins> // waitForCacheCount blocks until the given cache store has the desired number // of items in it. This will return an error if the condition is not met after a // 10 second timeout.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: panicked with "invalid memory address or nil pointer dereference", resulting in crash-looping. What you expected to happen: to gracefully handle nil-valued variables. How to reproduce it (as minimally and precisely as possible): Create a object with a field that, when passed to , returns . Anything else we need to know?: This is the offending code: The nil pointer dereference occurred on line 473. Environment: Kubernetes version (use ): 1.17.14 Cloud provider or hardware configuration: GKE OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig apps
/cc /triage accepted
/assign


Question: auditapis "k8s.io/apiserver/pkg/apis/audit" "k8s.io/apiserver/pkg/audit" "k8s.io/apiserver/pkg/endpoints/handlers/negotiation" <ins> "k8s.io/apiserver/pkg/registry/rest" </ins> "k8s.io/utils/pointer" )

In a cluster with large enough numbers of objects, it seems to be impossible to clean them up with . Because the deletecollection is failing while listing the objects from etcd. The failure happens after exactly 34 seconds, due to the 34-second max timeout introduced in PR (cc for insight on that PR). The deletecollection should be able to run for longer, and succeed. I recognise that there are still risks of the API server using too much RAM, if deleting a very large collection due to issue . But, until that issue is fixed, it seems like allowing to run for longer might be a good idea. As far as I can tell, the 34 second timeout does not apply to normal LIST calls, so on that basis I feel it should not apply to the giant unchunked LIST call that happens inside . a few hundred thousand objects of some particular type. (Argo workflowtaskresults happen to be the type I'm looking at while I write this) deletecollection. (Optionally passing some kind of fieldSelector, but I don't think that's essential) that a 504 failure code is logged in kube-audit logs, and that a log entry similar to this is logged by the API Server (In practice, the total time may be a little over ms). far as I can tell, the delete collection really has failed at this point. I notice that PR says that, in general, "The go-routine can keep running, while the timeout logic will return a timeout to the client" but I don't think that's what's happening here. I think that, because the 34 sec timeout is used in the context of the etcd call, the etcd list call really does fail, and so the code never proceeds to actually deleting anything. No response 22 AKS <details</details<details</details<details</details<details</details
/sig scalability
/cc
the deadline has been in effect before PR was introduced If we remove this deadline from here, then the deletecollection request will have at most to complete (default). The list call that is failing in your case, is a direct call to the storage, right? on the contrary, we don't enforce the timeout for LIST thinking out loud here: I don't know if it makes sense to remove the from should we allow timeout for DELETECOLLECTION to exceed ? thoughts?

The list call that is failing in your case, is a direct call to the storage, right? It reads from etcd, not from API Server Cache. Is that what you were asking Looking at the bigger picture, I think your question about 60s timeout is a good one. Especially in the context of this: In particular, changing the timeouts won't solve all of the problems with DELETECOLLECTION. My assumption, when logging this issue, is that it might be a quick way to solve just some of them.
I think we should remove this - it's effectively a regression that we introduced with the PR. That's a good question, and I'm not entirely sure. Maybe we should, but I would do that separately. In the meantime, I would much more prefer to switch the DeleteCollection implementation to internally use pagination, as it would allow us to 1) not block on full list (and as a result make some progress) 2) would be some preparation towards potentially exposing a Limit option in the API... /kind regression
/sig api-machinery
/triage accepted
I suggest the following: - can you please take care of reverting the 34s timeout for DeleteCollection calls? this is done, let's close this issue and continue operating on
sounds good, thanks! /assign


Question: TestSampleAPIServer(f, aggrclient, imageutils.GetE2EImage(imageutils.APIServer)) }) <del> /* Release: v1.25 Testname: Aggregator, manage lifecycle of an APIService Description: An APIService is created which MUST succeed. The APIService status when replaced MUST succeed. Given the updating of the APIService status, the fields MUST equal the new values. The APIService status when patched MUST succeed. Given the patching of the APIService status, the fields MUST equal the new values. The APIService when replaced MUST succeed. Given the updating of the APIService, the fields MUST equal the new values. It MUST succeed at deleting a collection of APIServices via a label selector. */ framework.ConformanceIt("should manage the lifecycle of an APIService", func() { </del> <ins> ginkgo.It("should manage the lifecycle of an APIService [Serial][Disruptive]", func() { </ins> ns := f.Namespace.Name framework.Logf("ns: %v", ns)

pull-kubernetes-e2e-kind [sig-storage] CSI .* It seems that happens since 7/8 The test timeout the docker daemon is killed, but the process keeps running, but it fails to do anything because all the features are missing, in this case, the port forwarding from the host to the apiserver container, as we can see in the following log entry As consequence, all commands fail, like the ones that gather the logs or clean up the environment The sig storage test take more than 10 minutes we shouldn't have test with that duration on presubmit, those also should be tagged as slow ~The trigger may be related to the ginkgo v2 update, the failures start to happen the same day that merged~ /sig testing /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc I know this people is in vacation, ccing just for visibility /cc
I don't know if correlation implies causality here, but it seems to start the day that the ginkgov2 was merged /cc is there any change in behavior for which the tests keep running, I remember some discussion about this between you and about the failure modes
hey there correlation doesn't necessarily imply causation - but it's definitely the first place to start looking ;) there shouldn't be any explicit changes in V2's behavior here however there were changes around how Ginkgo captures stdout/stderr during parallel test runs (i.e. when running with ) that, in some circumstances involving external processes that hang, can cause to hang. Often when this happens Ginkgo catches it and prints out a message but I assume you haven't seen anything like that? Here's something to try: If: (a) these specs are running in parallel (b) the failure is somewhat reproducible Then: try running with to see if the flaky failure is resolved. If the test consistently exits without hanging with this flag on then that would point to this output-interception issue and potentially help us figure out the next step for debugging things.

This sort of failure would also happen in Ginkgo v1 but it would be intermittent. V2 has a much cleaner implementation of output interception which has the side-effect of making this sort of failure more consistent. usually the issue ends up being with a spawned process holding on to an open pipe or failing to clean up somehow.... it can be a gnarly thing to debug :(
it seems that the CSI tests are taking into account the time to delete the namespace, per example takes 10 mins, starts at and takes 5 mins deleting the namespace it feels like this is related on how CSI tests are structured, made some comments about this here it seems that is the problem, the sig-storage test take the time to delete namespace into account , maybe for the way the use the hooks for AfterEach, can this be refactored, the current CI presubmits are very flake because of this problem
we are at a success 47% rate on presubmits, /priority critical-urgent
Hi I'm not sure I understand the issue. Is there a change in cleanup behavior to cause these tests to start timing out? cc who also looked into storage test cleanup in the past.
Yeah, it seems to be related to the ginkgo v2 migration, there were some behaviour changes ... I'm just trying to get some help because I couldn't find the problem and I am lost on those CSI e2e test logic
i put this in the sig-testing slack but in case it's relevant here: one thing to note is that in ginkgo v1 the —timeout flag used to apply to each suite separately. So if you are running 3 suites with a timeout of 50 minutes each suite gets 50 minutes. In v2 the timeout governs the entire test run - that means all 3 suites must finish within 50 minutes. not sure if that is in play here (are there multiple suites running?)
let's try this theory
can you please take on the slow tests from sig-storage that are running on presubmits?

I can do it but I'm afraid of breaking something, you can see them here
This seems to have improved
nope, it didn't improve, it may be an artifact of not having many PR or low load on the CI, who knows
hey -- looking at just that last run I'm seeing: 13:18 is the first timestamp I see. Eventually (~around 13:35) the specs start running. Everything is happy and several hundred specs run in parallel and pass. The last three specs then fail - these three are running in parallel. The first spec is: [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand [AfterEach] should resize volume when PVC is edited while pod is using it and has an error trying to delete the test namespace: The seems to wrap it in a try and so this timeout/failure doesn't mark the spec as failed. The storage specific cleanup keeps running... This stuff: appears to succeed - but then this step: seems to hang. Eventually, at 14:18:32 I see: Presumably after three minutes of polling the namespace hasn't been removed. But now that the outer container/infrastructure has timed out the tcp connection attempting to continue polling fails. It looks like that 1 hour timeout is governed by the test setup, nto the Ginkgo test suite. With the container/infrasturcture gone, the spec reports the error "". but this too is wrapped in a try. The spec finally fails in the with: again, because the container/infrasturcture was killed at 14:18:32. This failure now bubbles up as the failure associated with the spec. But in reality the underlying issue seems to be a failure to delete the namespace. The second spec is: [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral [AfterEach] should create read-only inline ephemeral volume Here too we see: ^ this appears to be the actual failure. Eventually the reports a failure at 14:18 after the container/infrastructure times out.

Finally, the third spec is: [sig-storage] CSI mock volume [AfterEach] Delegate FSGroup to CSI driver [LinuxOnly] should not pass FSGroup to CSI driver if it is set in pod and driver supports VOLUMEMOUNTGROUP here, once again, we see ^ this appears to be the actual failure. As with the other specs, eventually the reports a failure at 14:18 after the container/infrastructure times out. So, it looks like something_ happens around 14:08 that puts the cluster in a bad state and these volume-related namespaces time out trying to delete. This pushes the run time for the entire suite (including the setup bits before Ginkgo runs) past the 1 hour limit imposed by the container/infrastructure layer. I'm not sure this is a V1 vs V2 issue (perhaps some previously implicit behavior aroudn test ordering has changed - but these parallel specs basically run the same way in V1 as in V2). Have there been any changes to the storage stuff introduced around the same time as the V2 upgrade? Perhaps this is an infrastructure resource/exhaustion issue - 700 specs managed to succeed before these last three failed!
I don't know if the custom cleanup and the aftereach are conflicting and hanging
no, i don't believe so. they are running sequentially for a given spec.
we do some crazy things, :smile: We need to minimize the diff to find out the root cause, and we can't have the presubmit jobs with a 45% success rate, I've reverted the AfterEach change and I'm asking sig-storage to tag the slow tests accordenly to current policies (5 mins) The lead now is in:
I am attempting to do some bisection... Here's a job from 7/5/22 where all of the csi tests are finishing in under 3 minutes:
Here's a later run with quick running times: I think I need to be looking at periodics though, not pull jobs
here's the earliest run I found with slow times:
Looking at CI jobs: Here's the first job where tests started taking 10 mins: The run before that was fine:
nicely done ! how does one figure out which commit has those two runs correspond to?

Good run was based on the following commit: Slow run was on the following commit: Not a lot of commits between the two and none of them look particularly suspicious
Towards the beginning of the test log, it shows the kubernetes version used to create the kind cluster:
any changes merge to test-infra around the range between those runs (and maybe slightly before)?
not really:
kind versions also look the same
BTW here's the prow job history page starting on the last good run: There is a very clear step up in test time after that.
I think next step is to try to locally rerun the tests against the good commit and the two commits after the good run to see if we can rule them out.
Cross posting from find the report attached with the logs for successful tests I've checked Test starts It seems the problem is on the namespace deletion The storage tests add a custom hook to delete the namespace that seems that is the one causing they take longer
The storage tests run all these cleanups inside the test context has namespace deletion changed somehow ? any possible idea on how ginkgov2 could affect this workflow?
I run this test repeatedly on my desktop, the test passed every time with around 50 seconds to finish the test. And I did some random and destructive testing - terminate the test execution in the middle of process, and hit the situation that namespace deletion is hanging, I check the log and found that is because there is a pod in that namespace, and both the namespace and pod are hanging at the phase of . And the termination never finish unless I force to remove them. Here is the log I can see from kubelet, show that has already been created. I suspect the hang we saw should hit the situation similarly, something in that namespace, pod? prevent us to delete the namespaces successfully. Current code doesn't dump the resource in that namespace since the namespace deletion is done in the , here is the code for that. So, is that possible we could hit the similar situation when all of the testcases run together? we could be able to dump the everything in that namespace when the termination is hanging. will dump the logs from each components, such as controller manger which might give us some hints on this issue, but I don't know where to get those logs.
The reason why the CSI tests manage namespace deletion is because of deletion ordering issues.

Each test installs its own CSI driver, however that CSI driver has to be deleted last otherwise Pods will fail to teardown and PVCs will fail to delete due to an uninstalled CSI driver.
make sense, but the log says the is not found. but I do find that csi driver is presented in that cluster, what am I missing here?
the and stuff in these tests is complex and i do wonder why the code isn't calling and instead. nonetheless i would expect this code as written to work: the will always run before the returns to its caller and Ginkgo v1 and v2 both basically guarantee that for a given process no two s will run in parallel - so all those closure variables shouldn't see any pollution from one test to the other within a given process. but. there's a lot of code here and while i don't see any obvious places where test pollution is creeping in due to these shared variables i can't say i'm 100% confident. also - while i don't yet see how ginkgo v2 vs v1 could be causing this we definitely haven't ruled anything out yet! one question: where are these namespace integers being generated (eg ) are we confident that the other parallel processes will never also pick 2296? or is it possible that multiple processes are picking the same integer and so cross-process pollution is occurring? it could be a good use of time run the suite while focusing just these CSI specs and running them in parallel with verbosity turned on so we can see what's going on. my sense is the current hypothesis is that a passing test is inadvertently deleting a resources that another test running in parallel is using thereby causing it to run slower. since we don't run with in verbose mode we won't see the relevant logs from the passing test. that would be my suggestion for a next step. run just these CSI specs in parallel with and see if we learn more from the logs.
I think I found the root cause. I did some debugging on my own PR, and debugged the GCE run because the kind run does not capture kube-controller-manager logs: test log: kube-controller- The commit that I thought wasn't suspicious is actually suspicious:
cc please help take a look. I think the new tests are disruptive and causing namespace deletion in other tests to fail.
Great to hear that!

Can you pls share us how to achieve that if we want to check the log from those components? where is the log file I can get?
For example from my test run above, this is where the control plane logs are: It would be nice to collect this from the kind jobs too.
~ (sorry) :) It's historical reason, my pr here: already commented on that, further actions are needed, this is also the action left in We just need time to get familiar with csi e2e test code and then start the migration. It's generated here: , and used here: In theory, I think this is possible, but the how the probability could be. Good suggestion, I think we can add this parameter along with the to the test code, and make it masked with comment. Anyway, thanks for the root causing!
the info is there, but the failing jobs doesn't have it because the job times out cleaning the kind cluster and the related logs If you take a working job the controller-manager logs are in and this confirms theory and demonstrates once more that correlation doesn't imply causality my fault :(, sorry all for the noise
another observation that, once this test started to run, it caused the job to flake !
great job getting to the bottom of this !
no failures since the offending test was reverted ! confirming msau theory
that is itself a bug, and shouldn't be happening. we should get and handle a TERM from prow well before we get killed and the TERM handler should be capturing logs.
log collection in timed out kind e2e runs should be fixed now , , sorry about that


Question: # Install aufs kernel module apt-get install --yes linux-image-extra-$(uname -r) <ins> # Install aufs tools apt-get install --yes aufs-tools </ins> DOCKER_OPTS="${DOCKER_OPTS} -s aufs" elif [[ ${docker_storage} == "devicemapper" ]]; then DOCKER_OPTS="${DOCKER_OPTS} -s devicemapper"

On AWS with Vivid, auplink is not installed. There are messages in the docker log about auplink not being in the path. Inspection of the format-disks script confirms this to be the case. (PR coming soon, just opening an issue so we have something to track it back to!)


Question: // service or not.  If this field is specified when creating a Service // which does not need it, creation will fail. This field will be wiped // when updating a Service to no longer need it (e.g. changing type). <ins> // This field cannot be updated once set. </ins> // +optional optional int32 healthCheckNodePort = 12;

We are currently using v1.19 and it seems that if you have a service with set to then changing the health check node port is not supported. One way to get around this is by changing the to and then changing it back to in addition to specifying the with the desired port number. This behavior is not very well documented and the below error message is not very useful: I wonder if we could also add another case below to release the current health check port and allow changing it to the user specified value given it's valid? It's possible that after creating a someone wants to change it to a user-provided and the current way to do it is only by toggling the . This feature with the documentation would be more straightforward way to make any port changes.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
19 is EOL, last supported version is 1.22 It seems the message has changed since then, this area had important changes in latest releases that improved the API behavior However, the field still can not be updated once set, what you do switching the is wiping the value and reallocating it. It is a documentation problem then
IMO, the amount of effort it would take to make this mutable is not justified when there is a workaround. Convince me the workaround doesn't work? Overall HCNP is a buried implementation detail.
I can't speak much to the amount of effort, but I can speak a bit to the value: a user, I don't expect healthcheckNodePort to be immutable. I'm not sure if my mental model is too simplistic, but it feels like a part of the configuration I control, and I can change NodePorts, so why not the healthcheckNodePort. the work around isn't necessarily obvious. It also takes experimenting with to know that it'll even work. this in a production environment is not trivial.

If changing the healthcheckNodePort was supported then you could change the config to the state you want it to be, and your rollout orchestration tooling goes and applies it to your collection of k8s clusters. With the work around you now have to have a 2 step process. After step 1 of this process your service is using a traffic policy which might break your application (say you depend on client ip). So to avoid the downtime you'd have to build some reasonably off-pattern orchestration that doesn't just apply the configuration from the commit you used for your release, but instead two commits, or two configs, etc. Operationally this isn't actually that cheap to build, especially as this is likely to be a one-off operation and not something you do every release. Not exactly the same thing, but related to healthcheckNodePort, I've also found that specifying it explicitly vs not explicitly in the config, and switching between the two modes, has unexpected behavior as well. If you started out as having it specified, applying a config with it unspecified tries to allocate a new value, and complains that it can't change healthcheckNodePort for traffic policy local. Why not just use the already allocated value? Going from unallocated to specifying it explicitly at the same value seems to works fine.
The latter issue should be fixed in recent releases - what version are you seeing that on? I understand it would be more consistent to be mutable but there are real downstream implications to changing it, code complexity, test complexity, etc, and I don't understand why you want to change it in the first place. Humans should not be paying attention to that field. It should not even be in spec, but for historical reasons.
Same version as original post, v1.19. Glad to hear that issue is fixed in later versions! We're making steady progress updating, but not all caught up yet. I don't want to insist on this fix, as I said I'm not aware of the fix complexity and leave it up to you guys to balance that out, just wanted to provide the value perspective. As for not being exposed to users at all, I think we'd probably have been better off if we never knew about it.

Unfortunately our original configs had it explicitly set, eventually ran into conflicts in various k8s clusters and tried to transition to having it be unspecified, and thus ran into the above situation.
Yeah, there's really no good reason to set it, pretty much ever, and lots of reasons not to. I will cross-check that it is fixed if I get a moment tomorrow.
It has been fixed by you Tim
I thought so, I just wanted to triple check :)


Question: _ "k8s.io/kubernetes/pkg/apis/certificates/install" _ "k8s.io/kubernetes/pkg/apis/coordination/install" _ "k8s.io/kubernetes/pkg/apis/core/install" <ins> _ "k8s.io/kubernetes/pkg/apis/discovery/install" </ins> _ "k8s.io/kubernetes/pkg/apis/events/install" _ "k8s.io/kubernetes/pkg/apis/extensions/install" <ins> _ "k8s.io/kubernetes/pkg/apis/flowcontrol/install" _ "k8s.io/kubernetes/pkg/apis/imagepolicy/install" _ "k8s.io/kubernetes/pkg/apis/networking/install" _ "k8s.io/kubernetes/pkg/apis/node/install" </ins> _ "k8s.io/kubernetes/pkg/apis/policy/install" _ "k8s.io/kubernetes/pkg/apis/rbac/install" _ "k8s.io/kubernetes/pkg/apis/scheduling/install"

What happened: Tried to convert a v1beta1 Ingress object to v1, and received the following error: What you expected to happen: The conversion to work. How to reproduce it (as minimally and precisely as possible): Then run: /sig cli /assign
Can i work on this issue?
/assign
/triage accepted


Question: <ins> /* Copyright 2016 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package net import ( "net" "testing" ) func getIPNet(cidr string) *net.IPNet { _, ipnet, _ := net.ParseCIDR(cidr) return ipnet } func TestIPNetEqual(t *testing.T) { testCases := []struct { ipnet1 *net.IPNet ipnet2 *net.IPNet expect bool }{ //null case { getIPNet("10.0.0.1/24"), getIPNet(""), false, }, { getIPNet("10.0.0.0/24"), getIPNet("10.0.0.0/24"), true, }, { getIPNet("10.0.0.0/24"), getIPNet("10.0.0.1/24"), true, }, { getIPNet("10.0.0.0/25"), getIPNet("10.0.0.0/24"), false, }, { getIPNet("10.0.1.0/24"), getIPNet("10.0.0.0/24"), false, }, } for _, tc := range testCases { if tc.expect != IPNetEqual(tc.ipnet1, tc.ipnet2) { t.Errorf("Expect equality of %s and %s be to %v", tc.ipnet1.String(), tc.ipnet2.String(), tc.expect) } } } </ins>

This is at least partially responsible for - some tests can't run when dns is down. Logs from just before the crash: It got killed because Kubelet decided to kill it (readiness probe failed): It started happening quite recently - I'm trying to locate a build which caused this. Repro: run "Kubelet should not restart containers across restart" test - it caused this behavior for me multiple times today. cc - I applied the P0/v1.3 label/milestone, as I think it should be a blocker. Feel free to decrease the priority if you disagree
gce-serial 1511 seems to be the first one with this problem.
Or 1510 - hard to tell, as it was a broken run. There's no PR around that looks too suspicious though:/
I locally reverted and it seems that DNS is no longer failing. Sadly I have to run to catch a train, but maybe someone can pick it up here ?
Taking a look now. The PR should be unrelated...I am checking to see if the Federation DNS changes went in yet. Add nethealth container to prepull manifest
I search the restart counts for jobs from 1510-latest and gce-serial Job 1540 is the first one to show a non-zero kubedns restart count. If there is a commit responsible, it would be between 1539 and 1540 (June 21 7 AM -11 AM) 1539 end of run 1540 end of run Latest
The kubedns:1.3 container hasn't been updated since (Sat, 04 Jun 2016 03:29:37.847 GMT). GET
cc The kubedns container code is from June 04th. Dawn suggested that its networking issues that are causing kubedns container to fail its readiness check. I will need to add logic to print kubedns container logs during e2e tests, I am trying to search for any logs that might help, otherwise I will have to debug the next run live.
As reported, a simple kubelet restart triggers this issue - it looks like 10.0.0.1 is unreachable from the kubedns container.

The kubedns container blocks indefinitely waiting for a connection to the kubernetes service at 10.0.0.1 before setting up the healthz handlers - the readiness probe will fail until kubedns successfully connects to the API server.
- you can try reverting and running it again - it worked for me for some reason (it might be by accident though)
I think the root cause is that cbr0 doesn't have an IP address assigned after kubelet restart - all the pods running behind cbr0 are isolated from the network. Not only kube-dns, but after my kubelet restart, the other addon pods are also stuck in a crash loop.
This sounds like a blocker even more now.
Looks like a regression. I am investigating.
Found the root cause. There is a subnet ip mismatch. vs . Hence kubenet thinks they are different and clears the address. Will submit a fix soon.
i found coredns status is CrashLoopBackOff and i check the logs : i got tips from so i restart my docker , i check the logs , there is no error in logs again


Question: srcs = [ "credentialmanager_test.go", "vsphere_test.go", <ins> "vsphere_util_test.go", </ins> ], embed = [":go_default_library"], deps = [

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Provsioning a PVC using a storage class with vSphere as the cloud provider failed with error "runtime error: index out of range". What you expected to happen: The PVC should have been created. How to reproduce it (as minimally and precisely as possible): Create a PVC using a storage class with vSphere as the cloud provider and a VMware vCenter Server where the default language is not English. Anything else we need to know?: K8s in conjunction with vSphere PVC relies on English error messages from the VMware API in order to determine the canonical path for a newly created volume (VMDK file). The faulting code (v1.11.3, still the same on master) can be found here (starting at line 312): The line "canonicalVolumePath = match[1]" (340) in the above code is the problem because "File (.*?) was not found" (English) never matches for a different language (e.g. German: "Datei [DATASTORE] kubevols/kube- wurde nicht gefunden") and a “runtime error: index out of range” oocurs in the "kube-controller-manager". This causes the already created volume (VMDK file) not to be reported back to the K8s system which in turn will try to recreate it repeatedly in a loop (fails with "Cannot complete the operation because the file or folder [DATASTORE] /kubevols/kubernetes-dynamic-pvc-[UUID] already exists"). Find below the corresponding log excerpt (kube-controller-manager): Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): n/a Kernel (e.g. ): n/a Install tools: n/a Others: n/a <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign


Question: } // translate ip string to IP for _, ipStr := range nodeAddress.UnsortedList() { <del> ips = append(ips, net.ParseIP(ipStr)) </del> <ins> a := net.ParseIP(ipStr) if a.IsLoopback() { continue } ips = append(ips, a) </ins> } return ips, nil }

What happened: For services with type NodePort (and LoadBalancer) ipvs entries for localhost (ipv4 and ipv6) are created. Here is an example for a service with so the same nodeport is used for ipv4 and ipv6; The localhost entries can't be used and most likely they never will. What you expected to happen: Since "::1" most likely will never work and 127.0.0.1 will not work without an unlikely Linux kernel update these entries must not be created. Further, since the use of localhost addresses for nodeports is the only know reason to require (), this requirement shall be removed for proxy-mode=ipvs. How to reproduce it (as minimally and precisely as possible): Start a cluster with proxy-mode=ipvs and some services Do; Anything else we need to know?: That ipv6 access to [::1]:nodeport does not work (and most likely never will) is discussed in For 127.0.0.1 is is a bit more complicated as discussed in , with the excellent deep-dive into the kernel by in This may seem like a minor issue but clean-up of useless things is always good. I also have (vague) plans to propose improvements for the address handling for proxy-mode=ipvs and the localhost entries becomes an unwanted problem. If this issue is accepted as a bug I'll be happy to do a PR. Environment: Kubernetes version (use ): all supporting proxy-mode=ipvs Cloud provider or hardware configuration: any OS (e.g: ): any Kernel (e.g. ): any Install tools: none Network plugin and version (if this is a network-related bug): any Others: /kind cleanup /sig network /area ipvs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
test in ipv6 + iptables , ::1 is not work for NodePort. /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/area kube-proxy
/remove lifecycle-stale
/remove-lifecycle stale <sigh...
Since it is highly unlikely that the kernel will be updated to allow this for ipv4 and it will never work for ipv6 I would really want to remove this from the ipvs proxier.
Do you think I can go ahead and disable the setup for localhost node-ports in the ipvs proxier? Since it have never worked it would not be a user facing change. Or should we discuss it on the sig/network meeting?
I personally think that this was implicitly discussed and confirmed here , it is an implementation detail of IPVS and a known issue that never worked in IPVS, so it will be safe to disable directly without further discussion. However, I defer to for a final decision /assign
I made a PR. ~It's just 3 lines~ Now 8 lines. I forgot the config setting.


Question: // When the last pending initializer is removed, and no failing result is set, the initializers // struct will be set to nil and the object is considered as initialized and visible to all // clients. <ins> // +patchMergeKey=name // +patchStrategy=merge </ins> repeated Initializer pending = 1; // If result is set with the Failure field, the object will be persisted to storage and then deleted,

There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery


Question: if s.AdvertiseAddress == nil || s.AdvertiseAddress.IsUnspecified() { hostIP, err := insecure.DefaultExternalAddress() if err != nil { <del> return fmt.Errorf("Unable to find suitable network address.error='%v'. "+ "Try to set the AdvertiseAddress directly or provide a valid BindAddress to fix this.", err) </del> <ins> return fmt.Errorf("unable to find suitable network address.error='%v'. "+ "Try to set the AdvertiseAddress directly or provide a valid BindAddress to fix this", err) </ins> } s.AdvertiseAddress = hostIP }

Currently as authentication and authorization are both standard config for all clusters, the kubeapiserver insecure http handler is unsecure, which means the cluster is vulnerable to the attack. So it should be deprecated, even removed totally.
/sig api-machinery
I think removing the insecure option is a good idea. Can you send an email the sig-apimachinery mailing list with a proposed timeline? I think this should be like other GA features and have a one year clock.
Sure, I will send it ASAP.
:+1: