Question: apiVersion: apps/v1 kind: Deployment metadata: <del> name: metrics-server-v0.4.4 </del> <ins> name: metrics-server-v0.5.0 </ins> namespace: kube-system labels: k8s-app: metrics-server addonmanager.kubernetes.io/mode: Reconcile <del> version: v0.4.4 </del> <ins> version: v0.5.0 </ins> spec: selector: matchLabels: k8s-app: metrics-server <del> version: v0.4.4 </del> <ins> version: v0.5.0 </ins> template: metadata: name: metrics-server labels: k8s-app: metrics-server <del> version: v0.4.4 </del> <ins> version: v0.5.0 </ins> spec: securityContext: seccompProfile:

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI -- (gce-ubuntu-master-default) Kubernetes e2e suite: BeforeSuite (There are other flakes in this job, but they are failing less often and different test each time, so let's say this issue is only for BeforeSuite) is not starting: Spyglass: Triage: Mentioned in /cc <!-- Please see the deflaking doc () for more guidance! --
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
/cc Since it is related to the metrics server. the metrics server keeps dying as the log shown in I cannot get more logs of metrics servers from the CI artifacts. Currently, we are using metric server 0.4.4 and the latest 0.5.0 is released. Are there some known bugs that may be related to this test failure?
I don't think so. /cc
The metrics-server fails to bind ~because there is another process listening~ on port 443 are some of the konnectivity pods listening on the port :443?
In the dockerfile of metrics-server if has ? Maybe it has permission problem?
ahh, right, nice catch, I didn't pay attention to the error and blindly assumed collision :sweat_smile: ... who can take an action on that? has this changed recently?
If the metrics-server version used after , I think it is correct. has no change recently.

I vaguely remember some changes recently to reduce privileges on pods on CI, tthis seems likely a permission problem, does this ring a bell? EDIT This PR introduced the change , any special reason why we want to run it in a privileged port if this is accessed using a Service? EDIT 2 let see if this is the problem
Yeah, Maybe you can refer to this, the release notes about v0.4.4 metrics-server . I think you have resolved this problem.
Release v0.3.X was already using privileged port 443, as v0.4.X has migrated to distroless image we needed to introduce to give it needed privileges. What's weird for me is that the test is flaking and not failing. Could it be related to node where MS is scheduled?
You mean, there may be other reasons? If so, may require in-depth analysis.
As far as I can tell, is never starting properly, even when the jobs are successful. If we were to take this job, that has succeeded: From the serial logs, we can see that the metrics-server pod is crash looping: But from the build logs, we can see that at some point the pod was deemed ready: To me it seems that there are 2 issues, one being that metrics-server doesn't have the capabilities it needs to run and another one that still needs to be investigated where metrics-server is sometimes marked ready even though it is in a crash looping state.
This is a problem with the metrics-server deployment on the CI jobs and the metrics tests I don't know which sig owns these components but it will be better to find first an owner, what is is clear is that the metrics tests are not reliable since once you set a readiness probe on the metrics-server it fails consistently , , however the tests pass cc
it seems the last try was the good one :smile:


Question: "clientConfig":      "ClientConfig defines how to communicate with the hook. Required", "rules":             "Rules describes what operations on what resources/subresources the webhook cares about. The webhook cares about an operation if it matches _any_ Rule.", "failurePolicy":     "FailurePolicy defines how unrecognized errors from the admission endpoint are handled - allowed values are Ignore or Fail. Defaults to Ignore.", <del> "namespaceSelector": "NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is other cluster scoped resource, it is not subjected to the webhook.nnFor example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as follows: "namespaceSelector": {n  "matchExpressions": [n    {n      "key": "runlevel",n      "operator": "NotIn",n      "values": [n        "0",n        "1"n      ]n    }n  ]n}nnIf instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {n  "matchExpressions": [n    {n      "key": "environment",n      "operator": "In",n      "values": [n        "prod",n        "staging"n      ]n    }n  ]n}nnSee https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ for more examples of label selectors.nnDefault to the empty LabelSelector, which matches everything.", </del> <ins> "namespaceSelector": "NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook.nnFor example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as follows: "namespaceSelector": {n  "matchExpressions": [n    {n      "key": "runlevel",n      "operator": "NotIn",n      "values": [n        "0",n        "1"n      ]n    }n  ]n}nnIf instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {n  "matchExpressions": [n    {n      "key": "environment",n      "operator": "In",n      "values": [n        "prod",n        "staging"n      ]n    }n  ]n}nnSee https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ for more examples of label selectors.nnDefault to the empty LabelSelector, which matches everything.", </ins> } func (Webhook) SwaggerDoc() map[string]string {

If I am not mistaken, means that cluster-scoped resources other than namespaces are broken with admission webhooks. There is even a unit test checking that:
To clear any ambiguity around the word "broken", the current code behavior is that most cluster-scoped resources are not subjected to any webhooks. I wrote them like this because I was concerned with the potential bootstrapping deadlocks: a webhook that enacts policies for cluster-scoped resources might cause bootstrapping deadlocks for the cluster or for its own deployment. Thinking out loud, perhaps for cluster-scoped resources, the apiserver should only call the webhooks if the resources' labels match the , so that the critical bootstrapping resources can be excluded from the webhooks, thus breaking the bootstrapping deadlocks. This aligns with what we do for (see ). The downside is that the field name "namespaceSelector" becomes inappropriate.
I think we still need namespaceSelectors. We don't want to label pods. But I agree we actually want another objectLabelSelector. For namespaces these two coincide, i.e. we only have to define how they are applied if both are set or we forbid both at the same time. I tend to the later.
raised up a security issue regarding . A user can change labels to skip webhooks. doesn't have this issue because a user usually doesn't have permission to change the namespace's labels.
Users normally cannot create cluster-scoped objects either. But techincally it's a valid point. On the other hand, my main use-case are CRDs. They are not critical to the cluster health (maybe self-hosted clusters are different with etcd operator), i.e. you normally cannot shoot yourself into the foot with a webhook on them. For CRDs we do not need a selector at all. In other words: IMO we should enable webhooks for cluster-scoped objects, even if we have no in place.
Sounds reasonable. I'll create a PR ()
We would like to have the ability to apply the admission controller on specific namespaces only, and not on cluster level resources. This is useful if the list of possible resource Kinds that we want to apply the controller for is large, and we want to limit when it runs as much as possible to avoid affecting the whole cluster in case there's an issue.

I think this can be done by adding a SkipClusterResources flag (or something similar) in addition to the NamespaceSelector, to the Webhook struct. What do you think?
I think adding a flag is reasonable. It doesn't have the security issue as the idea does (). The expressiveness of is limited compared to , e..g., admins cannot use to run webhooks on some cluster-level resources while skipping others. However it's not a big drawback, because admins can specify more detailed GVK to limit the scope of webhook in that case.
Zeroth thought: s/SkipClusterResources/scope=[clusterboth]/g First thought: can't you just exclude cluster scoped resources, or include only namespace scoped resources in the webhook definition? Second thought: can a namespace selector be crafted that excludes the empty string?
0: Are you sure that's cleaner? The namespace selector picks which namespaces to use, and skipCluster decides whether to include cluster resources or not. Not setting it maintains current behavior (skipCluster=false) and whether cluster level resources are included is very clear. scope=cluster/namespace/both will require us to default to "both" which I find slightly more confusing. Also what to do if I specify cluster AND include namespace selector - is that an invalid spec or do we ignore the selector? 1: Yes. This is the current best thing we can do. If the list of resources is large though (for example, if I am implementing an alternative resource quota controller) then this solution requires us to maintain and update a long list of resources. This may be ok, since as long as our controller can't support certain types of resources it doesn't make sense to watch for them... 2: Doesn't look like it - the current code specifically checks if the attribute is a cluster level resource and skips the entire label matching process.
Re: 0: fair point, but let's at least name it in the positive: matchClusterScopedResources=true|false or something like that.
I proposed it in the negative to make it backwards compatible, so that "false" keeps current behavior. I agree that negatively named booleans are not not bad. I'll treat it as true if unspecified and use matchClusterScopedResources.
In today's semantic, such a namespace selector will exclude namespaced objects whose namespace has no label.


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/client/record" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/healthz" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/master/ports" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" "github.com/GoogleCloudPlatform/kubernetes/plugin/pkg/scheduler"

Found this by calling the "/validate" end point on the master: root curl 127.0.0.1:8080/validate <snip{ "component": "scheduler", "health": "failure", "healthCode": 1, "msg": "404 page not foundn", "err": "unhealthy http status code: 404 (404 Not Found)" }, </snipSeems like any requests to it return a 404: root netstat -p -l | grep kube-scheduler tcp 0 0 localhost: : LISTEN 4105/kube-scheduler root curl 127.0.0.1:/healthz 404 page not found root curl 127.0.0.1: 404 page not found
I'm not seeing this on my 0.14.1 cluster: $ curl http://127.0.0.1:8080/validate { "component": "scheduler", "health": "success", "msg": "ok", "err": "nil" }, $ curl http://127.0.0.1:/healthz ok
This is happening on HEAD. seems to have caused this. /cc
Could we rollback this PR?
is this only on the scheduler?
Yes. The change basically broke the /healthz mapping in kube-scheduler, so the health pings are failing. The "validate-" doesn't hit it, but the master's "/validate" does. This is causing the internal cluster validation checks to fail.
GKE relies on the /validate endpoint on the master (since it can't run the validate- script) and this has broken our GKE-CI build since it was merged this morning.
I think we can revert the PR and let profiling break for now. what do you think?
SGTM, sorry for the breakage.
Ahh, you guys already revert the commit? The fix should be very simple. Let me send PR shortly to fix this.
Thanks for the fix and sorry for the breakage


Question: ToDest      = "--to-destination " Recent      = "recent " MatchSet    = "--match-set " <ins> SrcType     = "--src-type " </ins> ) type Rule map[string]string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: This is discussed on When externalTrafficPolicy: Local is used, in GKE, external traffic coming in to the Load Balancer works correctly. But traffic that originates within the GKE cluster is broken, a TCP SYN is not responded to. My use case: I run gitlab, gitlab-runner, docker-registry, wiki inside kubernetes. But I also access the docker-registry outside kubernetes. So my hostname for it is ''. In DNS this resolves to the LB IP. Actually, all 3 services have DNS names that resolve to the same LB IP, which then goes to the nginx ingress. so , , are generally addressed as LB-IP w/ HOSTNAME+SNI. Its 'astonishing' to not be able to access this DNS name when I'm 'inside' but can when I'm 'outside'. If I enable the IP transparency (because in my wiki I want the IP of the user who made the edit), then my gitlab-runner fails(since it cannot pull from the registry). Tim Hockin has this to say: and then: What you expected to happen: My expectation is that it should not matter where my traffic originates, if it is to a globally-routeable public IP it should work. How to reproduce it (as minimally and precisely as possible): Configure a Load Balancer, an ingress, and 2 services. Have service 2 talk to service one using a DNS name that maps to the Load Balancer. Anything else we need to know?

: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.2-gke.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-05-10T17:23:18Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Google GKE OS (e.g. from /etc/os-release): BUILDID=.89.0 NAME="Container-Optimized OS" KERNELCOMMITID= GOOGLECRASHID=Lakitu VERSIONID=66 PRETTYNAME="Container-Optimized OS from Google" VERSION=66 GOOGLEMETRICSPRODUCTID=26 HOMEURL="" ID=cos Kernel (e.g. ): Linux gke-k8s-default-pool--d3ft 4.14.22+ SMP Thu May 10 17:54:42 PDT 2018 x8664 Intel(R) Xeon(R) CPU @ 2.00GHz GenuineIntel GNU/Linux Install tools: Others:
/sig network
Agree that this is weird and feels left-out. The fix is not complicated but I need an owner to carry it through testing.
if nobody is working on it can I take it as my first issue?.
Is anybody working on this issue. Seems like ipvs might have the same problem.
/assign
I believe I'm running into this problem and I think adding as Tim mentioned would fix it. are you working on this? If not I don't mind taking a stab at it :)
Thanks, Sorry I kinda forgot this thing. I was fixed this thing just like you said But encounter some problem when testing in gce environment, which caused by route rules. Sorry I forgot the detail, can you do some test in your environment?
It seems the pull request is stalled on this. Does anyone have any input?
Being relied on by istio
Following this.
Did this fix get lost in the shuffle?

Any chance getting this merged in? This also impacts Digital Ocean's Kubernetes + LB setups out of the box.
As far as I know it hasn't
Merge On Mon, Feb 18, 2019, 7:58 PM Jarred Capellman wrote:
still affecting me on gke. i want to enable the so that I can use the spam-blocking in wordpress (which is by ext ip), but I also want to refer to other services which may be internal to the cluster via their proper name (to get the `tls etc).
Same here.
I have more or less spelled out the fix, but I don't have teh bandwidth to polish it off right now. I am looking for volunteers...
I can take this given I've run into this a while back /assign
I was only able to reproduce this when trying to reach the public IP from the host network. Reaching the public IP from inside the pod network is working (as per this ). It's possible I missed a step in trying to reproduce this but wanted to confirm with you first. Are you seeing this for traffic originating only from host network or does it also apply for pods in the cluster network?
Hi. My situation. I have a DNS name (git.MYDOMAIN). It runs on a self-hosted gitlab via nginx-ingress via LoadBalancer in GKE. Other pods of mine sometimes reference https://git.MYDOMAIN. They may happen to run in the same cluster as git.MYDOMAIN, they may run in a different cluster. When they run in the same cluster on GKE, they fail if is enabled. Specifically, since the DNS resolves to (My external Loadbalancer IP) an IP outside of GKE, it goes out, hits the LB, and then somewhere along the reverse path is lost. I have calico enabled (as per GKE managed). I am not using Istio. I am using a cluster that predates the new private IP ones. I have not tried disabling the LoadBalancer and just using node ip for the ingress. I have not tried to see if there is any difference if the Pod trying to reach my git is on the same node as it or not.

Based on the proxy rules (may be different at this point cause I'm looking at master), I don't think your traffic is even reaching the LB, it's just being dropped by the iptable rules on the host. My original question was, are the pods trying to reach the same-cluster external IP have , if yes, can you reproduce on pods not on host network?
I do not have set.
Are you still on v1.10?
this is the most recent version on gke afaik.
I opened a fix here based on the comments and some testing I did on my GKE cluster. It needs more validation though
Again, I wasn't able to reproduce this for pods that are within the cluster CIDR (but I could on the host network). The rule that allows for pods inside the cluster CIDR to still reach pods behind an LB is this: Are you able to check if a rule like ^ exists on your cluster?
Here's an example, right now (www is hosted on ingress w/ loadbalancer w/ local policy). Note: this example is from the node (not a pod).
Here's the full rules:
^ can you show me the same test from within a pod (not host network)? I have a fix for within a node but I'd like to make sure it's reproducible inside a pod before I dig deeper
Interestingly its working in a Pod on this cluster now. (?!) I guess some previous upgrade of k8s on gke has altered it? on cannot reach on on can reach. hmmm.
ok w/o the node being able to reach this i'm still blocked (since my container registry might run there, and thus kubelet cannot reach, so my gitlab CI pipeline cannot reach). but, i guess, now the pods w/o hostnetwork can, so some progress
Thanks for checking should fix the internal reachability issue from the host. Though it requires SNAT.
Hello Maybe better way is to check if and then skip to KUBE-SERVICES chain? So, request will be sent to loadbalancer and source IP will be preserved. Can you try this?
Solution described above will affect traffic from pods, it will not be . Then maybe we can replace with something to send traffic to loadbalancer?

I think this will break existing implementations, the fwchain is what routes the LB IPs to the correct service chain. Not all LB implementations (think MetalLB) depend on node port for routing
I tried to exclude node IP with from on node without local endpoints and it works! What do you think about this solution? I tested with:
I believe there's a similar rule for that already but it doesn't check source IP If you jump to FW chain, you end up in XLB chain which drops traffic if there's no local endpoints. Does your test also work on nodes where there's no local endpoints? Can you also verify that your traffic actually goes to the LB and is not being masqueraded through the service chain?
Yes, I just source IP check to this rule on node without local endpoints, so local traffic doesn't jump to FW, it doesn't drops in XLB and it goes to LB. Of course, if you do this on node with local endpoints you will not reach that endpoint (at least when using Metallb). This solution doesn't work if nodes haven't network access to LB. Your solution haven't this problem. For me it is more important then saving source IP, so now I think your solution is better. But why you add jump to MARK-MASQ before jump to SVC when each SEP chain already contains jump to MARK-MASQ? I tested your solution with:
Good point, I think might be redundant, will update the PR soon :)


Question: "reflect" "testing" <ins> "github.com/google/uuid" </ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/kubernetes/pkg/scheduler/framework" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Below are the failures while running unit test with golang in master branch. Reference : $ go version go version devel go1.17- Thu Apr 22 01:40:02 2021 +0000 linux/ppc64le Note: Occurs in x86 architecture as well. The tests should Pass as with stable go git clone cd kubernetes make test KUBERACE=-race WHAT=./pkg/scheduler/framework/runtime/ Master branch of kubernetes repo. OS: Linux Observed failure on ppc64le and x86 architectures Below is the attached log:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I request help on getting this triaged. Is there a procedure to follow to get this accepted as a bug?
The Go compiler developer has identified the bugs in the test that is causing it to fail. They are listed in the last comment in the Go issue before closing it.
So the correct way to compare function would be something like would you test it in your env?
The issue still exists. Can you please reopen?
I am not sure if you can access this link for our CI job. Can you please check? https://prow.ppc64le- The above tests have been failing every day in our jobs!
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
downloads go from url="${GOLANGVERSION}.linux-" downloads go from "curl -X GET http://build-?project=build_commit -o " I suspect the postsubmit-master-golang-kubernetes-unit-test-ppc64le failed for wrong golang version.

failed for other problems. The TestMerge is not failed. For ppc64le failures, there are several issues in golang: (fixed yesterday)
Yes the go versions for both the jobs are different and it is intentional: : It is meant to run Kubernetes unit tests with the stable version of golang : It is meant to run Kubernetes unit tests with golang built from the master branch of As I mentioned in the description of this issue, these unit test cases are failing only on golang that is built from master branch and passing with stable versions of golang. And as Lynn mentioned in , the golang team has concluded that these failures are due to the bugs in the test case. And the golang issues that you mentioned do not seem to be related to the failures mentioned in this issue.
With , is false Per , do you mean that we should change to ?
No. In general comparing is the right thing to do. But if the Value is actually a , which is not comparable, comparing will panic.
/sig scheduling
Note from golang team about this bug is here:


Question: return num } <ins> func countEndpointsSlicesNum(epList *discoveryv1.EndpointSliceList) int { // EndpointSlices can contain the same address on multiple Slices addresses := sets.Set[string]{} for _, epSlice := range epList.Items { for _, ep := range epSlice.Endpoints { if len(ep.Addresses) > 0 { addresses.Insert(ep.Addresses[0]) } } } return addresses.Len() } </ins> // restclientConfig returns a config holds the information needed to build connection to kubernetes clusters. func restclientConfig(kubeContext string) (*clientcmdapi.Config, error) { Logf(">>> kubeConfig: %s", TestContext.KubeConfig)

Kubernetes e2e suite: [It] [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should reject everything except leases The job was run on Aug 5th. I had a PR fixing typos in comments of kubelet source code. It is not related to api-machinery ipv6 but the test failed. The re-test passed. No response No response It seems the test is recently . /sig api-machinery /sig testing
It is failing considerably , it seems it fails to connect to the webhook that matches the time the test was I don't fully understand the test but it seems that is interacting with the cluster, I don't have clear if the deny policy affects everything and this is causing the race with the pod that implements the webhook /priority critical-urgent /release blocker /cc
/assign
cc
Debugged with Joe offline, paste for reference click open stderr on the failure to get the timeline
still failing after merge of
/triage accepted
It's suspicious that of all the admission webhook tests that start up a webhook pod, this is the one that flakes. I'll look more closely.
Looks like the webhook in this test may be configured to deny requests except leases globally in the cluster once enabled., which might create a race where, if the webhook is registered first , the controller setting up the webhook pod is denied requests by the webhook. I'll scope the webhook matching to the test namespace per offline suggestion.
Please let me know if I can be of help debugging this.Have free cycle and would like to contribute.
/assign
I can still see references to the expression that gives error Example run: Could you help confirm that the tests are completely removed? Thank you. cc
I think this is another test that is impacting the rest of the tests. Probably because the tests are running in parrallel. We might want to scope things down. Currently working on a fix.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
this is a new bug, please review all the test that were and verify they are scoped to the custom created namespaces by the test framework, and if not, please tag them as cc:
/cc
I assume this could be closed with merged? /cc
Looks like the exact flak as reported in this issue has been resolved. /close
Closing this issue. <detailsIn response to : Looks like the exact flak as reported in this issue has been resolved. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package v1 import ( "io" "net/http" "strings" "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes/scheme" restfake "k8s.io/client-go/rest/fake" ) func TestCreateWithEventNamespace(t *testing.T) { event := &v1.Event{ ObjectMeta: metav1.ObjectMeta{ Name:      "test", Namespace: "default", }, } cli := &restfake.RESTClient{ Client: restfake.CreateHTTPClient(func(request *http.Request) (*http.Response, error) { resp := &http.Response{ StatusCode: http.StatusOK, Body:       io.NopCloser(strings.NewReader("{}")), } return resp, nil }), NegotiatedSerializer: scheme.Codecs.WithoutConversion(), } tests := []struct { name    string ns      string event   *v1.Event wantErr bool }{ { name:  "create event", ns:    "default", event: event, }, { name:    "create event with different namespace", ns:      "other", event:   event, wantErr: true, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { e := newEvents(New(cli), tt.ns) _, err := e.CreateWithEventNamespace(tt.event) if (err != nil) != tt.wantErr { t.Errorf("CreateWithEventNamespace() error = %v, wantErr %v", err, tt.wantErr) return } }) } } func TestPatchWithEventNamespace(t *testing.T) { event := &v1.Event{ ObjectMeta: metav1.ObjectMeta{ Name:      "test", Namespace: "default", }, } cli := &restfake.RESTClient{ Client: restfake.CreateHTTPClient(func(request *http.Request) (*http.Response, error) { resp := &http.Response{ StatusCode: http.StatusOK, Body:       io.NopCloser(strings.NewReader("{}")), } return resp, nil }), NegotiatedSerializer: scheme.Codecs.WithoutConversion(), } tests := []struct { name    string ns      string event   *v1.Event data    []byte wantErr bool }{ { name:  "patch event", ns:    "default", event: event, data:  []byte{}, }, { name:    "patch event with different namespace", ns:      "other", event:   event, data:    []byte{}, wantErr: true, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { e := newEvents(New(cli), tt.ns) _, err := e.PatchWithEventNamespace(tt.event, tt.data) if (err != nil) != tt.wantErr { t.Errorf("PatchWithEventNamespace() error = %v, wantErr %v", err, tt.wantErr) return } }) } } func TestUpdateWithEventNamespace(t *testing.T) { event := &v1.Event{ ObjectMeta: metav1.ObjectMeta{ Name:      "test", Namespace: "default", }, } cli := &restfake.RESTClient{ Client: restfake.CreateHTTPClient(func(request *http.Request) (*http.Response, error) { resp := &http.Response{ StatusCode: http.StatusOK, Body:       io.NopCloser(strings.NewReader("{}")), } return resp, nil }), NegotiatedSerializer: scheme.Codecs.WithoutConversion(), } tests := []struct { name    string ns      string event   *v1.Event wantErr bool }{ { name:  "patch event", ns:    "default", event: event, }, { name:    "patch event with different namespace", ns:      "other", event:   event, wantErr: true, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { e := newEvents(New(cli), tt.ns) _, err := e.UpdateWithEventNamespace(tt.event) if (err != nil) != tt.wantErr { t.Errorf("UpdateWithEventNamespace() error = %v, wantErr %v", err, tt.wantErr) return } }) } } </ins>

Follow up from We should cover events expansion - both for core v1: events v1beta: with unit tests - can you please take care of it?
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign
GitHub didn't allow me to assign the following users: try, I, want, to, have, a. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign I want to have a try Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: klog.V(4).InfoS("Using secondary node IP", "IP", secondaryNodeIP.String()) } <del> if externalCloudProvider { </del> <ins> if externalCloudProvider || cloud != nil { // Annotate the Node object with nodeIP for external cloud provider. // // We do this even when external CCM is not configured to cover a situation // during migration from legacy to external CCM: when CCM is running the // node controller in the cluster but kubelet is still running the in-tree // provider. Adding this annotation in all cases ensures that while // Addresses flap between the competing controllers, they at least flap // consistently. // // We do not add the annotation in the case where there is no cloud // controller at all, as we don't expect to migrate these clusters to use an // external CCM. </ins> if nodeIPSpecified { if node.ObjectMeta.Annotations == nil { node.ObjectMeta.Annotations = make(map[string]string) } node.ObjectMeta.Annotations[cloudproviderapi.AnnotationAlphaProvidedIPAddr] = nodeIP.String() } <ins> } if externalCloudProvider { </ins> // If --cloud-provider=external and node address is already set, // then we return early because provider set addresses should take precedence. // Otherwise, we try to look up the node IP and let the cloud provider override it later

I am reproducing this in OpenShift using the OpenStack cloud provider, but nothing here is specific to either OpenShift or OpenStack. When migrating a cluster which uses in-tree cloud provider to use external cloud-provider, we use the following process: cloud controllers in KCM CCM kubelets with --cloud-provider=external Between steps 1&2 the node-lifecycle, routes, and service controllers are not running anywhere, but this situation isn't unsafe for the cluster and we can keep it short. Between steps 2&3 is more interesting. The node controller is running in CCM, and the kubelets are running with --cloud-provider=openstack (or whatever, just not external). The main loop of node-controller will ignore a newly created node because . However, the loop does not have a similar exclusion. . Note that we shortcut if the kubelet is using an external cloud provider, but not if we don't. So between steps 2&3 both kubelet and CCM's node controller are managing Node.Addresses. If the legacy cloud provider and external CCM don't return the same list of NodeAddresses this causes flapping. Running OpenShift on OpenStack we have 2 problems. Firstly, our legacy cloud provider and external CCM produce different lists. We don't think this would be a problem in practise, except for the second problem. Kubelet adds the annotation to the node containing the node IP. This allows the CCM node controller to apply the same filtering based on node IP as kubelet does for the legacy cloud provider. This means that, although the lists are different, the node IP is still listed first which is the only thing that matters in practise. The problem is that kubelet only adds the annotation when --cloud-provider=external which, between steps 2&3, it is not. This means that Node.Addresses flaps between a correct and a broken configuration, as the NodeAddresses returned by the external CCM cannot be filtered by node IP without the annotation. The ideal solution here would be to ensure that only 1 controller is managing Node.Addresses during this period in the same way we do for Node creation with the external CCM taint. This would fix the issue entirely.

A fix with a much smaller scope would be to have kubelet add the node IP annotation unconditionally so, although Node.Address can potentially flap, it will at least be between 2 valid configurations. Given that this should only ever be a problem during the upgrade window, the smaller fix may be more expedient. Node.Addresses should remain valid at all times when kubelet is configured with --cloud-provider=openstack but the CCM is also running. Good luck No response 24 OpenStack (but as described, not OpenStack-specific) No response No response No response No response
/sig cloud-provider
we've an agenda item for next week's (11 May) SIG cloud provider meeting about this topic
/triage accepted
Would it work to swap steps 2 and 3?
Good question. I don't think so. That would leave us without the Node, Node lifecycle, and Service controllers until all kubelets were updated. This would mean: No cluster scale-up or scale-down No creation, deletion, or modification to LB Services I think we could handle this in OpenShift, although it's a bit fragile and I suspect updating all the kubelets can take a really long time on a big cluster. However, I think this would break cluster-api or any deployment tooling which rolls out updates by creating new machines. I think this would never complete: the new nodes would never come up because the node controller isn't running.
I am still trying to understand the issues here while reviewing your pr: , but I agreed with your comments above: That would leave us without the Node, Node lifecycle, and Service controllers until all kubelets were updated. This would mean:


Question: rgName := az.getLoadBalancerResourceGroup() allLBs, rerr := az.LoadBalancerClient.List(ctx, rgName) if rerr != nil { <ins> if rerr.IsNotFound() { return nil, nil } </ins> az.Event(service, v1.EventTypeWarning, "ListLoadBalancers", rerr.Error().Error()) klog.Errorf("LoadBalancerClient.List(%v) failure with err=%v", rgName, rerr) return nil, rerr.Error()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I recall that this issue was reported previously in It was fixed in the master back then with and it was cherry-picked into with However we still observe the same issue with the legacy azure cloud-controller-manager v1.19.9. legacy azure cloud controller-manager to be able to delete LoadBalancer when the underlying resource group is deleted. a LoadBalancer manually the underlying resource group the LoadBalancer created in step 1 sure that the deletion of the LoadBalancer fails Events of the Service: Kubernetes version (use ): v1.19.9 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure /cc
Thanks for reporting the issue. It should be a bug inside cloud provider. /triage accepted


Question: if err := nodeController.monitorNodeStatus(); err != nil { t.Errorf("%v: unexpected error: %v", item.description, err) } <del> // Give some time for rate-limiter to reload time.Sleep(500 * time.Millisecond) </del> for zone, state := range item.expectedFollowingStates { if state != nodeController.zoneStates[zone] { t.Errorf("%v: Unexpected zone state: %v: %v instead %v", item.description, zone, nodeController.zoneStates[zone], state) } } <del> zones := testutil.GetZones(fakeNodeHandler) for _, zone := range zones { // Time for rate-limiter reloading per node. time.Sleep(50 * time.Millisecond) nodeController.zonePodEvictor[zone].Try(func(value TimedValue) (bool, time.Duration) { uid, _ := value.UID.(string) deletePods(fakeNodeHandler, nodeController.recorder, value.Value, uid, nodeController.daemonSetStore) return true, 0 }) } podEvicted := false for _, action := range fakeNodeHandler.Actions() { if action.GetVerb() == "delete" && action.GetResource().Resource == "pods" { podEvicted = true </del> <ins> var podEvicted bool start := time.Now() // Infinite loop, used for retrying in case ratelimiter fails to reload for Try function. // this breaks when we have the status that we need for test case or when we don't see the // intended result after 1 minute. for { podEvicted = nodeController.doEviction(fakeNodeHandler) if podEvicted == item.expectedEvictPods || time.Since(start) > 1*time.Minute { </ins> break } }

Running latest from master (currently commit ), I'm seeing TestMonitorNodeStatusEvictPodsWithDisruption is flakey: test -c ./pkg/controller/node true; do if ! ; then break; fi; done for failure I've narrowed it down , which is supposed to process all the items (nodes) in the queue. This queue has a capacity of 1, and at least in the test code, a refill rate of QPS. If the call to fails , it will stop processing wherever it happened to be in the queue. I've some extra logging locally, and when it fails, I see it processing nodes 1 and 2 before TryAccept fails. The issue is that the pod it's trying to evict is on node0, and it doesn't process node0 before TryAccept returns false. cc
I'm surprised we haven't hit this in jenkins yet
and removed the "leak" option referred to in the godoc, but it seems like it kept the leak behavior, not the retry behavior when the rate limiter is hit
Still seeing this issue
I didn't understand how this is related to sig/scheduling. Am I missing anything here?
guessing the flake is related to this line: anything on the order of milliseconds is going to be unreliable in testenvs that sometimes peg the CPU with parallel tests or it's related to the single item burst and the "abort when TryAccept fails" behavior
I can extend this sleep to something bigger, but I don't have cycles to debug it properly:/
any thoughts on that?
Generally we retry when TryAccept fails, but not in tests. So either we need to rewrite the test, or lengthen the sleep time. It's not perfect, but there's no way for "wait until rate limiter is reloaded".
I was working on this issue and I tried the change that you suggested and I notice that it fails with this change too() and I think rewriting the test with retry seems to be the option as I tried increasing 5000 ms and still it failed. I am new to the codebase, if there is piece of code where retry is available, I can check on that.
- what you wrote is concerning. I wish I had time to debug it:/ Frankly - those tests are old. Like really, really old. So they start to deserve a rewrite.

On the flip side - we'll need to rewrite them when making taint-based evictions default, so we'll need to allocate time for that anyway. If lengthening sleep time reduce the flakyness I vote for doing that, and just wait for upcoming rewrite.
having a delay in the loop before caused it to run for close 4 hours without terminating on 8GB,4 core machine. Is this something normal? I know, is not happy with this change but since we are rewriting this test anyways, this could be a work around that removes the flakyness. Let me know your thoughts. If this does work, can we remove the flake label and keep this issues as this is not a release blocker for 1.6?
- I don't understand which delay in which loop you mean. You a delay inside the "Try" call? What do you mean by 'run for close 4 hours without terminating'? That the test didn't fail? If so, then I'm all for such solution for stopping the bleeding.
, Yes I introduced a sleep in loop for zone, like Yes the test didn't fail but it didn't complete either even after 4 hours, so I am wondering if this causes timeout in testing infra.
It's strange indeed... Do you have any logs for the test by any chance? Adding this delay shouldn't be noticeable, as it just inserts some delay here: Sleeping under the lock is usually not the best idea, but 50 microseconds in test shouldn't make a lot of difference...
Unfortunately no, I can try reproducing this today. Probably by afternoon EST, I can have something, The way I am doing as of now is the way suggested by in the comments above. I can redirect that to a file and see. Let me know, if you are ok with approach.
I'm OK with anything that produces logs:) I generally do:
- out of curiosity. How often does this flake?
If I run my loop, it takes about 8-10 seconds to flake. , Marek Grabowski <:
I think I made a mistake. I did not take that while true loop out because of which it ran in infinite loop. I am sorry about that.

It means that with that delay introduced that I mentioned earlier it never failed even after running for 4 hours, which means we got rid of atleast test flakyness. if you can confirm if it is working, I will raise a PR which introduces this delay.
giving the ratelimiter time to refill inside the loop is consistent with the theory that is causing problems. If the rate limiter runs out of tokens in Try(), it returns with a comment, but the unit test does not try again later. I'm having a hard time following the code to know if non-test invocations have the same problem or if this is just a unit test issue.
The sleep addition seems to have helped. rgudimet <:
It's test only problem: Runs in , so it'll retry in the next round. I guess the problem here was actually introduces later than what suggests - I don't think that the removal of leaky behavior was problematic, but introducing a test that evicts more than one Node was. This means that if processing queue is fast enough, then rate limiter fails to reload, so adding the sleep there is actually a correct solution (as long as we don't have good rate limiter mock/testable rate limiter).
Thanks for confirmation. I will raise a PR for that now. As far as I understood, the issue is with unit test. If rate limiter fails to reload for a particular node, it cannot retry(unless the we separate out the logic which tells if the rate limiter has reloaded or not. - if we have that logic we can ask Try to run again by returning a false just like ).
Still an issue: https://k8s-
but not a blocker (the test is at fault)
Thanks, I will check this out after 1.6 release hopefully, with rewrites..
Has there been any recent incidents of this issue? It's marked for 1.7.
Closing - shows near nothing.
- this flakes out sporadically, I was able to reproduce this when debugging on my system, if I run it for long time.
https://k8s-


Question: } // createPodOrFail creates a pod with the specified containerPorts. <del> func createPodOrFail(f *framework.Framework, ns, name string, labels map[string]string, containerPorts []v1.ContainerPort) { </del> <ins> func createPodOrFail(f *framework.Framework, ns, name string, labels map[string]string, containerPorts []v1.ContainerPort, args ...string) { </ins> ginkgo.By(fmt.Sprintf("Creating pod %s in namespace %s", name, ns)) <del> pod := e2epod.NewAgnhostPod(ns, name, nil, nil, containerPorts) </del> <ins> pod := e2epod.NewAgnhostPod(ns, name, nil, nil, containerPorts, args...) </ins> pod.ObjectMeta.Labels = labels // Add a dummy environment variable to work around a docker issue. // https://github.com/docker/docker/issues/14203

The test only validates that the API objects are created E2e tests must verify the behaviour, i.e. that the Service and Endpoints actually work sending traffic. The file is not well organized and there is a lot of duplicate code, one possibility is to use current Services helper has a function to test the reachability that iterates over all service ports However, something that needs to be addressed is that the pods that are backing the service doesn't have any process listening on those ports
/help wanted /sig network /priority important-soon /triage accepted
I think that there are more tests that doesn't verify the reachability , those should be modified too /help-wanted
/help-wanted
/assign
/triage accepted
/assign /unassign I will take on this since it was more difficult than I expected, and is a Conformance test.. I assigned you to , that is similar to this, but creating a new test from scratch


Question: cmd.Flags().StringVar(&cfg.Networking.ServiceSubnet, "service-cidr", cfg.Networking.ServiceSubnet, "The range of IP address used for service VIPs") cmd.Flags().StringVar(&featureGatesString, "feature-gates", featureGatesString, "A set of key=value pairs that describe feature gates for various features. "+ "Options are:n"+strings.Join(features.KnownFeatures(&features.InitFeatureGates), "n")) <ins> cmd.Flags().Var(utilflag.NewMapStringString(&cfg.APIServerExtraArgs), "apiserver-extra-args", "A set of extra flags to pass to the API Server or override default ones in form of <flagname>=<value>") </ins> } if properties.use == "all" || properties.use == "controller-manager" { cmd.Flags().StringVar(&cfg.Networking.PodSubnet, "pod-network-cidr", cfg.Networking.PodSubnet, "The range of IP addresses used for the Pod network") <ins> cmd.Flags().Var(utilflag.NewMapStringString(&cfg.ControllerManagerExtraArgs), "controller-manager-extra-args", "A set of extra flags to pass to the Controller Manager or override default ones in form of <flagname>=<value>") } if properties.use == "all" || properties.use == "scheduler" { cmd.Flags().Var(utilflag.NewMapStringString(&cfg.SchedulerExtraArgs), "scheduler-extra-args", "A set of extra flags to pass to the Scheduler or override default ones in form of <flagname>=<value>") </ins> } cmd.Flags().StringVar(&cfgPath, "config", cfgPath, "Path to kubeadm config file (WARNING: Usage of a configuration file is experimental)")

/kind feature What happened: When calling , there is no way to override default flag values for the different components of the controlplane. This is surprising, as in the config structure (and in controlplane phase codebase itself), there are explicit settings for doing that (APIServerExtraArgs, ControllerManagerExtraArgs, SchedulerExtraArgs). What you expected to happen: Those config fields should be exposed as flags on and How to reproduce it (as minimally and precisely as possible): Try to deploy K8s using Kubeadm with a custom Admission Control setting (I'll follow up on this issue with a PR)
btw, I am not sure which sig owns kubeadm.
it's sig-cluster-lifecycle.
/sig sig-cluster-lifecycle
/sig cluster-lifecycle


Question: framework.ExpectEqual(statusUpdated.Status.Conditions[len(statusUpdated.Status.Conditions)-1].Message, "Updated by an e2e test", "The Message returned was %v", statusUpdated.Status.Conditions[0].Message) framework.Logf("Status.Condition: %#v", statusUpdated.Status.Conditions[len(statusUpdated.Status.Conditions)-1]) }) <ins> ginkgo.It("should apply an update to a Namespace", func() { var err error var updatedNamespace *v1.Namespace ns := f.Namespace.Name cs := f.ClientSet ginkgo.By(fmt.Sprintf("Updating Namespace %q", ns)) err = retry.RetryOnConflict(retry.DefaultRetry, func() error { updatedNamespace, err = cs.CoreV1().Namespaces().Get(context.TODO(), ns, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to get Namespace %q", ns) updatedNamespace.Labels[ns] = "updated" updatedNamespace, err = cs.CoreV1().Namespaces().Update(context.TODO(), updatedNamespace, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "failed to update Namespace: %q", ns) framework.ExpectEqual(updatedNamespace.ObjectMeta.Labels[ns], "updated", "Failed to update namespace %q. Current Labels: %#v", ns, updatedNamespace.Labels) framework.Logf("Namespace %q now has labels, %#v", ns, updatedNamespace.Labels) }) </ins> }) func unstructuredToNamespace(obj *unstructured.Unstructured) (*v1.Namespace, error) {

[X] Test approval issue: [X] Test PR: 27 Aug 2022 [x] Two weeks soak end date: 9 Sept 2022 [x] Test promotion PR: According to this APIsnoop query, there are still a remaining Namespace endpoint which is untested. replaceCoreV1Namespace Using a number of existing e2e test practices a new has been created for 1 Namespace endpoint. The e2e logs for this test are listed below. This query shows the following endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 1 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } debugLogThresholdsWithObservation("thresholds - reclaim not satisfied", thresholds, observations) <del> // determine the set of thresholds whose stats have been updated since the last sync thresholds = thresholdsUpdatedStats(thresholds, observations, m.lastObservations) debugLogThresholdsWithObservation("thresholds - updated stats", thresholds, observations) </del> // track when a threshold was first observed now := m.clock.Now() thresholdsFirstObservedAt := thresholdsFirstObservedAt(thresholds, m.thresholdsFirstObservedAt, now)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: unused images does not get deleted What you expected to happen: unused images get deleted How to reproduce it (as minimally and precisely as possible): configure kubelet with these args: Anything else we need to know?: when using above config, the node's events history showing: If I change to it works, node's events history showing these: The difference is there is an extra event, which is not seen with cofnig. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): Linux 192-168-1-1 4.4.0-63-generic -Ubuntu SMP Wed Feb 1 17:20:32 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig area/kubelet
Maybe we are with the same problem. Kubelet Parameters for tests: When we reach a disk consumption of 20% (for debug) - after 2 minutes, the deletion of logs, image docker, and then PODS should begin. But nothing happens. The debug logs kubelet saw the following: Problems are observed in the versions 1.6.X, 1.7.X In version 1.5.7, it looks like this: Can you help us? Or we help you, with Debug? Environment: Kubernetes version (use kubectl version): OS (e.g. from /etc/os-release): ubuntu 16.04.2 LTS Kernel (e.g. uname -a): Linux node2 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x8664 GNU/Linux Docker: Docker version 1.11.2, build b9f10c9_ PS We tried the rollback option from , but it did not change anything.
hello! Can you spare a little time to this problem?
Well, here is what we have found so far. In our setup, eviction.managerImpl.synchronize() frequently runs with a set of signalObservation objects, which contains the same data as in previous runs, even for "time" field. Probably this is due to slow cadvisor. Unfortunately, eviction.thresholdsUpdatedStats() considers such data expired and removes it from the list of observed thresholds. This was always the case, but before commit "time" field wasn't initialized, and thresholdsUpdatedStats allowed it to persists between iterations of synchronize() by explicit check for After neither check for zero, no check for one time being more recent then another, doesn't work as iterations are frequently performed with the same summary.Node.Fs object. And here we are with blinking "node conditions - observed: [DiskPressure]" in logs: one run of synchronize sets it, and next one discards it (runs are delimited with "synchronize housekeeping" message): The question is whether this logic in thresholdsUpdatedStats is correct now or not. Should garbage collection work even with slow cadvisor (if this is the reason of slow updates of summary.Node.Fs)?

Probably frequency of synchronize should be adjusted, or summary.Node.Fs updates should be synchronized with execution of synchronize()? We can correct the problem by monkey-patching makeSignalObservations and get rid of summary.Node.Fs.Time usage in signalObservation, but is it right?
The 2 minute grace period has not been met, so no action is taken. This is the difference between a "soft" and "hard" threshold: The soft only takes action after the threshold has been met for the specified period of time. If no new stats exist, the correct behavior is not to take action. This could delay action by 10 seconds, but it is better not to be overactive when evicting (we have had this problem in the past). The eviction manager assumes that disk pressure still exists when it does not have new stats. It basically just "skips" that synchronize call when that is the case.
dekhtyarev@ it looks like your node goes in and out of disk pressure. I see messages woven into your logs.
Hmmm, or maybe there is something here. Looks like the timer is reset for soft eviction thresholds.
No, On seeing this, I just ran several PODs on the node and did to achieve the conditions of eviction. I used ~22% of the disk, at the threshold of 80%. And disk consumption on the node did not change.
We just think that the timer resets, as the timestamp of the received data has not changed and this is taken into account in the eviction manager logic. Look at the field here: wrote about it, I think.
I see the issue. Ill try and get a bugfix into 1.8.
cc Soft Eviction thresholds have not been functional since 1.6...
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


[x] Original approval issue: [x] Two weeks soak end date: This issue is created to allow edit by /wip /hold This issue proposes the writing of a new test for the API endponts , , , with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this APIsnoop query, there are still some remaining Endpoints endpoints which are untested. a Service in a namespace all Endpoints in all Namespaces find the created Service(1) /update the Namespaced Endpoint for the Service(1) the Namespaced Endpoint for the Service(1) the Namespaced Endpoint for the Service(1) Namespaced Endpoint(1) via a Collection with a LabelSelector Discover useragents: If a test with these calls gets merged, Conformance coverage will go up by 4 points* /sig testing /sig architecture /area conformance
/reopen Wait for Promotion PR to Merge.
Reopened this issue. <detailsIn response to : /reopen Wait for Promotion PR to Merge. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } secondaryAllocatorCIDR := r.secondaryServiceIPs.CIDR() <del> if *(service.Spec.IPFamily) == api.IPv6Protocol && netutil.IsIPv6CIDR(&secondaryAllocatorCIDR) { </del> <ins> if (*(service.Spec.IPFamily) == api.IPv6Protocol) == netutil.IsIPv6CIDR(&secondaryAllocatorCIDR) { </ins> return r.secondaryServiceIPs }

What happened: When K8s is started in dual-stack mode with IPv6 as the "main" (or base) family services specified with ipFamily:IPv4 still get a ipv6 ; The is expicitly set to "10.0.0.0" using . What you expected to happen: Services with ipFamily:IPv4 shall get ipv4 ClusterIP addresses. How to reproduce it (as minimally and precisely as possible): Setup a K8s cluster in dual-stack mode with family IPv6 as base Define a service with ipFamily: IPv4 To make K8s think IPv6 is the main family the cue is to define ipv6 addresses to K8s binaries. Below are my settings as an example. I do not use but install "from scratch"; Master processes; is accessed with ipv4 which does not affect the K8s family. Worker processes; Anything else we need to know?: There are other problems with dual-stack/ipv6-base but may be caused by this one. For example does not work and some processes does not become "ready". Ipv6 traffic to ipv6 services and DNS works; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: - OS (e.g: ): BusyBox based own setup. Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): Others:
/sig network /cc
The problem seem to be; The problem occurs when an ipv6 range is specified first, even in an ipv4-based dual-stack cluster. But the ipv6 range must be specified first otherwise the "kubernetes" service gets an ipv4 address but an ipv6 backend; ~Perhaps the way to fix this problem is to make sure the service address is from the right family and require that the order to is "ipv4,ipv6".~ Bad idea!
There seem to be a option; But I can't see it in the help printout. I will try to use it...
:disappointed:
Here it where the kubernetes service address is set;
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: if regional && utilfeature.DefaultFeatureGate.Enabled(features.GCERegionalPersistentDisk) { disk, err = gce.getRegionalDiskByName(diskName) if err != nil { <del> glog.V(5).Infof("Could not find regional PD named %q to Attach. Will look for a zonal PD", diskName) err = nil } else { mc = newDiskMetricContextRegional("attach", gce.region) </del> <ins> return err </ins> } <del> } if disk == nil { </del> <ins> mc = newDiskMetricContextRegional("attach", gce.region) } else { </ins> disk, err = gce.getDiskByName(diskName, instance.Zone) if err != nil { return err

/kind bug If a PV's zone annotation format indicates that it's a regional PD, but pdName corresponds to a regular PD, currently the GCE PD plugin allows attach to succeed. It should fail, because the mismatch is often caused by user error. /sig gcp /sig storage /assign
Can we have an admission controller that checks for this user error?
how would an out of tree volume plugin enforce this restriction? Shouldn't we try to model surfacing the error using the flows available to CSI plugins, since the in-tree plugins are planned to migrate to that?
cc I think the ValidateVolumeCapabilities call may be able work for this purpose. We would still need some sort of CSI admission controller/webhook to make the callout to the plugin for this though. Another alternative. Today we already have an admission controller that auto-fills in zone information for GCE PDs. It is being moved into the cloud-specific admission controllers, and we could potentially extend this there.
Today the PVLabel admission controller already overwrites the user-specified label with the correct label, but it's important to have a secondary check in the plugin. Consider this scenario: both a regional and a regular disk exist with name , and the user's PV has the zone label matching the regional disk. The admission controller then retains the user specified label since it's valid. Next, the user deletes the regional disk, and the PV is left unchanged. When a pod is created with this PV, the regular disk will be attached, even though the attach should fail. As for CSI, yeah it looks like we could use as the call is designed for verifying pre-provisioned volumes. And I agree, if there are cloud-specific admission controllers in the future, it'd be good to include this check there instead. When CSI begins to support topology, ideally should also have a topology field. Synced with offline.


Question: framework.Failf("Unable to find v1alpha1.wardle.example.com in APIServiceList") } <ins> // As the APIService doesn't have any labels currently set we need to // set one so that we can select it later when we call deleteCollection ginkgo.By("Adding a label to the APIService") apiServiceName := "v1alpha1.wardle.example.com" apiServiceClient := aggrclient.ApiregistrationV1().APIServices() apiServiceLabel := map[string]string{"e2e-apiservice": "patched"} apiServiceLabelSelector := labels.SelectorFromSet(apiServiceLabel).String() apiServicePatch, err := json.Marshal(map[string]interface{}{ "metadata": map[string]interface{}{ "labels": apiServiceLabel, }, }) framework.ExpectNoError(err, "failed to Marshal APIService JSON patch") _, err = apiServiceClient.Patch(ctx, apiServiceName, types.StrategicMergePatchType, []byte(apiServicePatch), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch APIService") patchedApiService, err := apiServiceClient.Get(ctx, apiServiceName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to retrieve api service %s", apiServiceName) framework.Logf("APIService labels: %v", patchedApiService.Labels) ginkgo.By("Updating APIService Status") var updatedStatus, wardle *apiregistrationv1.APIService err = retry.RetryOnConflict(retry.DefaultRetry, func() error { var statusToUpdate *apiregistrationv1.APIService statusContent, err = restClient.Get(). AbsPath("/apis/apiregistration.k8s.io/v1/apiservices/v1alpha1.wardle.example.com/status"). SetHeader("Accept", "application/json").DoRaw(ctx) framework.ExpectNoError(err, "No response for .../apiservices/v1alpha1.wardle.example.com/status. Error: %v", err) err = json.Unmarshal([]byte(statusContent), &statusToUpdate) framework.ExpectNoError(err, "Failed to process statusContent: %v | err: %v ", string(statusContent), err) statusToUpdate.Status.Conditions = append(statusToUpdate.Status.Conditions, apiregistrationv1.APIServiceCondition{ Type:    "StatusUpdated", Status:  "True", Reason:  "E2E", Message: "Set from e2e test", }) updatedStatus, err = apiServiceClient.UpdateStatus(ctx, statusToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "Failed to update status. %v", err) framework.Logf("updatedStatus.Conditions: %#v", updatedStatus.Status.Conditions) ginkgo.By("Confirm that v1alpha1.wardle.example.com /status was updated") statusContent, err = restClient.Get(). AbsPath("/apis/apiregistration.k8s.io/v1/apiservices/v1alpha1.wardle.example.com/status"). SetHeader("Accept", "application/json").DoRaw(ctx) framework.ExpectNoError(err, "No response for .../apiservices/v1alpha1.wardle.example.com/status. Error: %v", err) err = json.Unmarshal([]byte(statusContent), &wardle) framework.ExpectNoError(err, "Failed to process statusContent: %v | err: %v ", string(statusContent), err) foundUpdatedStatusCondition := false for _, cond := range wardle.Status.Conditions { if cond.Type == "StatusUpdated" && cond.Reason == "E2E" && cond.Message == "Set from e2e test" { framework.Logf("Found APIService %v with Labels: %v & Condition: %v", wardle.ObjectMeta.Name, wardle.Labels, cond) foundUpdatedStatusCondition = true break } else { framework.Logf("Observed APIService %v with Labels: %v & Condition: %v", wardle.ObjectMeta.Name, wardle.Labels, cond) } } framework.ExpectEqual(foundUpdatedStatusCondition, true, "The updated status condition was not found. %#v", wardle.Status.Conditions) framework.Logf("Found updated status condition for %s", wardle.ObjectMeta.Name) ginkgo.By(fmt.Sprintf("Replace APIService %s", apiServiceName)) var updatedApiService *apiregistrationv1.APIService err = retry.RetryOnConflict(retry.DefaultRetry, func() error { currentApiService, err := apiServiceClient.Get(ctx, apiServiceName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to get APIService %s", apiServiceName) currentApiService.Labels = map[string]string{ apiServiceName: "updated", } updatedApiService, err = apiServiceClient.Update(ctx, currentApiService, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err) framework.ExpectEqual(updatedApiService.Labels[apiServiceName], "updated", "should have the updated label but have %q", updatedApiService.Labels[apiServiceName]) framework.Logf("Found updated apiService label for %q", apiServiceName) </ins> // kubectl delete flunder test-flunder <ins> ginkgo.By(fmt.Sprintf("Delete APIService %q", flunderName)) </ins> err = dynamicClient.Delete(ctx, flunderName, metav1.DeleteOptions{}) validateErrorWithDebugInfo(ctx, f, err, pods, "deleting flunders(%v) using dynamic client", unstructuredList.Items)

[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below.

This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted


Question: ) type AuditEvent struct { <del> Level             auditinternal.Level Stage             auditinternal.Stage RequestURI        string Verb              string Code              int32 User              string Resource          string Namespace         string RequestObject     bool ResponseObject    bool AuthorizeDecision string </del> <ins> Level              auditinternal.Level Stage              auditinternal.Stage RequestURI         string Verb               string Code               int32 User               string ImpersonatedUser   string ImpersonatedGroups string Resource           string Namespace          string RequestObject      bool ResponseObject     bool AuthorizeDecision  string </ins> } // Search the audit log for the expected audit lines.

We don't currently have any test coverage of impersonation for audit logging. I think the audit e2e test () would be a good place to do so. /area audit /area test /sig auth /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : We don't currently have any test coverage of impersonation for audit logging. I think the audit e2e test () would be a good place to do so. /area test /sig auth /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
assign I would like to work on this issue. Please give me some tips and instructions which i should take into consideration and will help me accomplishing this task successfully.
Take a look at the existing audit tests (linked in description), and the documentation on . I recommended doing a manual test first, so you understand how it works. The audit test actually already uses impersonation (), so you can just do something similar and then verify that it is logged correctly (the field).
Also, if you need help with the e2e framework, is the place to start.
Ok, thank you :)
GitHub didn't allow me to assign the following users: loqutus. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
How is it going? Do you need some help?
Just watch around.
I am pretty confused regarding how to write test cases
Can you give me more details about what it is you are struggling with? Writing test cases in general? I'm also new to the project but perhaps I can help you out. :)
Yes, I am struggling with writing test cases in general
Are you still struggling with test cases in general?

Please tell me if I am wrong, but I think here a test case is just testing that a particular call to the api logs specific information (represented by the struct AuditEvent). If I am correct, we don't need to create a complete new test case, but rather edit the exiting ones by checking one more info. Additionally, I am not sure that we can test this locally with kubetest, since the test seems to only accept the gce provider (google cloud engine). Hope this helps.
yes, I am struggling for writing test case as well as this was in my concious as well that how I am gonna test this test case
OK, I think I got what the issue is about. Please let me know if you are still struggling and prefer to hand it over. I'd like to give it a shot but don't want to steal the opportunity to make a first contribution to the project from you.
yaa sure you can give it a shot
FYI I'm working on this issue. I'm trying to get audit logging activated in minikube so I can manually test impersonation and see the log output (in order to see what the test has to cover). However, trying to get it activated in minikube is more difficult than I thought. :grimacing:
Hey how is it going with the manual test? The only way I found with minikube, is to start it with the following command (I use minikube v0.30.0). This activates audit logs, and configures the logs file path in the minikube's VM. Then you could use an api call using kubectl, with and without impersonation, and see the difference in logs. Notice that it does not matter if we have not created any pods at this point, because we just want to test the audit. To look into the logs you need to ssh into Minikube and look into a particular file. The manual test is a good thing to see how it works. But you don't actually need to code the whole thing, I think. as the current audit test already loads and parses the log file (see inside CheckAuditLines, and then parseAuditLine).
Hey , thanks for your extensive anser.

I managed to get in tested manually with minikube a couple of days ago, though it was not possible for me to get an costum audit policy file loaded even though it was available (somehow the extra config flag was ignored). Anyway, I spent the last couple of days figuring out how to best run the e2e tests on my macbook (I'm using Vagrant right now since there were some problems with running the tests in docker on mac). As you already mentioned in a previous comment, the audit tests only accept gce as a provider. I tried to simply comment that line out in order to see what happens if it runs on my local machine, but (unsuprisingly) it blew up. So the current status is, that I'm now trying to write the test case for impersonation relying on the CI to see if it runs through (since I cannot run the test locally) which is not ideal but I don't have a better idea atm.


Question: <del> /* Copyright 2015 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package network import ( "context" "fmt" "path/filepath" "strings" "time" "github.com/onsi/ginkgo/v2" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/wait" clientset "k8s.io/client-go/kubernetes" api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/test/e2e/feature" "k8s.io/kubernetes/test/e2e/framework" e2ekubectl "k8s.io/kubernetes/test/e2e/framework/kubectl" e2enetwork "k8s.io/kubernetes/test/e2e/framework/network" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eoutput "k8s.io/kubernetes/test/e2e/framework/pod/output" e2eresource "k8s.io/kubernetes/test/e2e/framework/resource" e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2etestfiles "k8s.io/kubernetes/test/e2e/framework/testfiles" "k8s.io/kubernetes/test/e2e/network/common" admissionapi "k8s.io/pod-security-admission/api" ) const ( dnsReadyTimeout = time.Minute // RespondingTimeout is how long to wait for a service to be responding. RespondingTimeout = 2 * time.Minute ) const queryDNSPythonTemplate string = ` import socket try: socket.gethostbyname('%s') print('ok') except: print('err')` var _ = common.SIGDescribe("ClusterDns", feature.Example, feature.NetworkingIPv4, func() { f := framework.NewDefaultFramework("cluster-dns") f.NamespacePodSecurityLevel = admissionapi.LevelPrivileged var c clientset.Interface ginkgo.BeforeEach(func() { c = f.ClientSet }) read := func(file string) string { data, err := e2etestfiles.Read(file) if err != nil { framework.Fail(err.Error()) } return string(data) } ginkgo.It("should create pod that uses dns", func(ctx context.Context) { // contrary to the example, this test does not use contexts, for simplicity // namespaces are passed directly. // Also, for simplicity, we don't use yamls with namespaces, but we // create testing namespaces instead. backendName := "dns-backend" frontendName := "dns-frontend" clusterDnsPath := "test/e2e/testing-manifests/cluster-dns" podOutput := "Hello World!" // we need two namespaces anyway, so let's forget about // the one created in BeforeEach and create two new ones. namespaces := []*v1.Namespace{nil, nil} for i := range namespaces { var err error namespaceName := fmt.Sprintf("dnsexample%d", i) namespaces[i], err = f.CreateNamespace(ctx, namespaceName, nil) framework.ExpectNoError(err, "failed to create namespace: %s", namespaceName) } for _, ns := range namespaces { e2ekubectl.RunKubectlOrDieInput(ns.Name, read(filepath.Join(clusterDnsPath, "dns-backend-rc.yaml")), "create", "-f", "-") } for _, ns := range namespaces { e2ekubectl.RunKubectlOrDieInput(ns.Name, read(filepath.Join(clusterDnsPath, "dns-backend-service.yaml")), "create", "-f", "-") } // wait for objects for _, ns := range namespaces { e2eresource.WaitForControlledPodsRunning(ctx, c, ns.Name, backendName, api.Kind("ReplicationController")) framework.ExpectNoError(e2enetwork.WaitForService(ctx, c, ns.Name, backendName, true, framework.Poll, framework.ServiceStartTimeout)) } // it is not enough that pods are running because they may be set to running, but // the application itself may have not been initialized. Just query the application. for _, ns := range namespaces { label := labels.SelectorFromSet(labels.Set(map[string]string{"name": backendName})) options := metav1.ListOptions{LabelSelector: label.String()} pods, err := c.CoreV1().Pods(ns.Name).List(ctx, options) framework.ExpectNoError(err, "failed to list pods in namespace: %s", ns.Name) err = e2epod.WaitForPodsResponding(ctx, c, ns.Name, backendName, false, 0, pods) framework.ExpectNoError(err, "waiting for all pods to respond") framework.Logf("found %d backend pods responding in namespace %s", len(pods.Items), ns.Name) err = waitForServiceResponding(ctx, c, ns.Name, backendName) framework.ExpectNoError(err, "waiting for the service to respond") } // Now another tricky part: // It may happen that the service name is not yet in DNS. // So if we start our pod, it will fail. We must make sure // the name is already resolvable. So let's try to query DNS from // the pod we have, until we find our service name. // This complicated code may be removed if the pod itself retried after // dns error or timeout. // This code is probably unnecessary, but let's stay on the safe side. label := labels.SelectorFromSet(labels.Set(map[string]string{"name": backendName})) options := metav1.ListOptions{LabelSelector: label.String()} pods, err := c.CoreV1().Pods(namespaces[0].Name).List(ctx, options) if err != nil || pods == nil || len(pods.Items) == 0 { framework.Failf("no running pods found") } podName := pods.Items[0].Name queryDNS := fmt.Sprintf(queryDNSPythonTemplate, backendName+"."+namespaces[0].Name) _, err = e2eoutput.LookForStringInPodExec(namespaces[0].Name, podName, []string{"python", "-c", queryDNS}, "ok", dnsReadyTimeout) framework.ExpectNoError(err, "waiting for output from pod exec") updatedPodYaml := strings.Replace(read(filepath.Join(clusterDnsPath, "dns-frontend-pod.yaml")), fmt.Sprintf("dns-backend.development.svc.%s", framework.TestContext.ClusterDNSDomain), fmt.Sprintf("dns-backend.%s.svc.%s", namespaces[0].Name, framework.TestContext.ClusterDNSDomain), 1) // create a pod in each namespace for _, ns := range namespaces { e2ekubectl.RunKubectlOrDieInput(ns.Name, updatedPodYaml, "create", "-f", "-") } // wait until the pods have been scheduler, i.e. are not Pending anymore. Remember // that we cannot wait for the pods to be running because our pods terminate by themselves. for _, ns := range namespaces { err := e2epod.WaitForPodNotPending(ctx, c, ns.Name, frontendName) framework.ExpectNoError(err) } // wait for pods to print their result for _, ns := range namespaces { _, err := e2eoutput.LookForStringInLog(ns.Name, frontendName, frontendName, podOutput, framework.PodStartTimeout) framework.ExpectNoError(err, "pod %s failed to print result in logs", frontendName) } }) }) // waitForServiceResponding waits for the service to be responding. func waitForServiceResponding(ctx context.Context, c clientset.Interface, ns, name string) error { ginkgo.By(fmt.Sprintf("trying to dial the service %s.%s via the proxy", ns, name)) return wait.PollUntilContextTimeout(ctx, framework.Poll, RespondingTimeout, true, func(ctx context.Context) (done bool, err error) { proxyRequest, errProxy := e2eservice.GetServicesProxyRequest(c, c.CoreV1().RESTClient().Get()) if errProxy != nil { framework.Logf("Failed to get services proxy request: %v:", errProxy) return false, nil } ctx, cancel := context.WithTimeout(ctx, framework.SingleCallTimeout) defer cancel() body, err := proxyRequest.Namespace(ns). Name(name). Do(ctx). Raw() if err != nil { if ctx.Err() != nil { framework.Failf("Failed to GET from service %s: %v", name, err) return true, err } framework.Logf("Failed to GET from service %s: %v:", name, err) return false, nil } got := string(body) if len(got) == 0 { framework.Logf("Service %s: expected non-empty response", name) return false, err // stop polling } framework.Logf("Service %s: found nonempty answer: %s", name, got) return true, nil }) } </del>

master-blocking: gce-cos-master-alpha-featuresKubernetes e2e suite:Failed runs: 09-16 09:58 CDT 09-15 23:41 CDT Both of these runs were on the same commit which also has passing builds. The test fails due to a , specifically when attempting to connect to the DNS backend service . The error indicates that the DNS backend service is either not running or not accessible on the specified address and port. This caused multiple retries in the test, with the pod failing to establish a connection to the backend service each time. As a result, the expected output "Hello World!" was not found in the logs, leading to the failure of the test. The service being unreachable could be due to several potential issues: misconfigured DNS, networking issues within the cluster, or the DNS backend service failing to start properly.Further investigation into the DNS service deployment and networking might be required to resolve this issue. There were two other failures ( , ) on the same commit that seemed to be related to cluster instance not being found: This caused failure when attempting to SSH into the instance and retrieve logs (kube-, , etc.). For those, the cluster setup (e2e-) likely failed, as no nodes were found, resulting in no instances being available. /sig network /kind flake cc:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /kind flake
This test should not be running in this job, it was picked recently ! since we already have coverage I think is better to remove it, is also using images that the sources are not available anymore


Question: } } }) <ins> // Regression test for https://issues.k8s.io/116925 ginkgo.It("should delete pods which are marked as terminal and have a deletion timestamp set after restart", func(ctx context.Context) { podName := "terminal-restart-pod" + string(uuid.NewUUID()) gracePeriod := int64(30) podSpec := e2epod.MustMixinRestrictedPodSecurity(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ TerminationGracePeriodSeconds: &gracePeriod, RestartPolicy:                 v1.RestartPolicyNever, Containers: []v1.Container{ { Name:    podName, Image:   imageutils.GetE2EImage(imageutils.BusyBox), Command: []string{"sh", "-c"}, Args: []string{` sleep 9999999 & PID=$! _term () { kill $PID echo "Caught SIGTERM!" } trap _term SIGTERM wait $PID trap - TERM # Wait for the long running sleep to exit wait $PID exit 0 `, }, }, }, }, }) ginkgo.By(fmt.Sprintf("Creating a pod (%v/%v) with restart policy: %v", f.Namespace.Name, podName, podSpec.Spec.RestartPolicy)) pod := e2epod.NewPodClient(f).Create(ctx, podSpec) ginkgo.By(fmt.Sprintf("Waiting for the pod (%v/%v) to be running", f.Namespace.Name, pod.Name)) err := e2epod.WaitForPodNameRunningInNamespace(ctx, f.ClientSet, pod.Name, f.Namespace.Name) framework.ExpectNoError(err, "Failed to await for the pod to be running: (%v/%v)", f.Namespace.Name, pod.Name) w := &cache.ListWatch{ WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { return f.ClientSet.CoreV1().Pods(f.Namespace.Name).Watch(ctx, options) }, } podsList, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).List(ctx, metav1.ListOptions{}) framework.ExpectNoError(err, "Failed to list pods in namespace: %s", f.Namespace.Name) ginkgo.By(fmt.Sprintf("Deleting the pod (%v/%v) to set a deletion timestamp", pod.Namespace, pod.Name)) time.Sleep(time.Second) err = e2epod.NewPodClient(f).Delete(ctx, pod.Name, metav1.DeleteOptions{GracePeriodSeconds: &gracePeriod}) framework.ExpectNoError(err, "Failed to delete the pod: %q", pod.Name) ctxUntil, cancel := context.WithTimeout(ctx, f.Timeouts.PodStart) defer cancel() ginkgo.By(fmt.Sprintf("Started watch for pod (%v/%v) to enter succeeded phase", pod.Namespace, pod.Name)) _, err = watchtools.Until(ctxUntil, podsList.ResourceVersion, w, func(event watch.Event) (bool, error) { if pod, ok := event.Object.(*v1.Pod); ok { found := pod.ObjectMeta.Name == podName && pod.ObjectMeta.Namespace == f.Namespace.Name && pod.Status.Phase == v1.PodSucceeded if !found { ginkgo.By(fmt.Sprintf("Observed Pod (%s/%s) in phase %v", pod.ObjectMeta.Namespace, pod.ObjectMeta.Name, pod.Status.Phase)) return false, nil } ginkgo.By(fmt.Sprintf("Found Pod (%s/%s) in phase %v", pod.ObjectMeta.Namespace, pod.ObjectMeta.Name, pod.Status.Phase)) return found, nil } ginkgo.By(fmt.Sprintf("Observed event: %+v", event.Object)) return false, nil }) ginkgo.By("Ended watch for pod entering succeeded phase") framework.ExpectNoError(err, "failed to see event that pod (%s/%s) enter succeeded phase: %v", pod.Namespace, pod.Name, err) // As soon as the pod enters succeeded phase (detected by the watch above); kill the kubelet. // This is a bit racy, but the goal is to stop the kubelet before the kubelet is able to delete the pod from the API-sever in order to repro https://issues.k8s.io/116925 ginkgo.By("Stopping the kubelet") startKubelet := stopKubelet() // wait until the kubelet health check will fail gomega.Eventually(ctx, func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, f.Timeouts.PodStart, f.Timeouts.Poll).Should(gomega.BeFalse()) ginkgo.By("Starting the kubelet") startKubelet() // wait until the kubelet health check will succeed gomega.Eventually(ctx, func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, f.Timeouts.PodStart, f.Timeouts.Poll).Should(gomega.BeTrue()) // Wait for the Kubelet to be ready. gomega.Eventually(ctx, func(ctx context.Context) bool { nodes, err := e2enode.TotalReady(ctx, f.ClientSet) framework.ExpectNoError(err) return nodes == 1 }, time.Minute, f.Timeouts.Poll).Should(gomega.BeTrue()) ginkgo.By(fmt.Sprintf("After the kubelet is restarted, verify the pod (%s/%s) is deleted by kubelet", pod.Namespace, pod.Name)) gomega.Eventually(ctx, func(ctx context.Context) error { return checkMirrorPodDisappear(ctx, f.ClientSet, pod.Name, pod.Namespace) }, f.Timeouts.PodDelete, f.Timeouts.Poll).Should(gomega.BeNil()) }) </ins> }) })

New TestGrid alerts at since commit/. Found 2 failing rows in node-kubelet-serial-containerd: 2 new, 0 other 2 new Test Failures in node-kubelet-serial-containerd. Test has failed 3 times since it last passed at commit/. It started failing between commit/ (Fri Mar 24 12:02 PDT) and commit/ (Fri Mar 24 16:03 PDT): . Test has failed 3 times since it last passed at commit/. It started failing between commit/ (Fri Mar 24 12:02 PDT) and commit/ (Fri Mar 24 16:03 PDT): . Not sure suspect No response /sig node release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake /remove-kind failing-test See
/remove-sig release
Taking a look at The test failed due to: so it means that the pod was not terminated and deleted properly in the test that ran prior. The is the one creating this pod, but I would expect that as part of the test teardown, the namespace is deleted (and thus the pod should be deleted).
The services log show that the kubelet never sent the request to delete the pod, that is strange...
From the build-logs, the test execution for when the pod was created: Kubelet logs for the pod - Full logs are here -
Here's the termination part of the logs:
It looks like to me the issue here is that the kubelet is being restarted in a bit of an unfortunate time resulting in at least two related issues: The pod status has been updated on the api-server to terminal just prior to the kubelet restarting. The status manager did not yet complete . Then after kubelet restarts and comes back up, it attempts to add this pod , (even though it is in terminal phase). This seems like the first issue: if the pod phase is terminal on the API server, we probably should not add it back.

The pod worker receives the , but the pod worker quickly terminates (). I am assuming this is because of , i.e. based on pod cache lookup the pod is terminal. The side affect of this is that the status manager never receives the signal that the pod is (which is normally done as part of . This is because in this case, we never call which sets to be true. As a result, I think will never return true, so this pod will never be able to be deleted.
I have made a test which will reproduce this behavior in a node e2e, see It can be reproduced by starting a pod and then deleting it and triggering kubelet restart at the right time. A watch is setup to trigger a kubelet restart as soon as the pod enters the succeeded phase. If the kubelet restart happens before the kubelet is able to delete the pod on the API server, but after the phase is updated, then we trigger the issue. The pod will be re- to kubelet after restart, but will never be deleted on the API server despite it having a deletion timestamp set and being in terminal phase. Here are logs from the and running the above repro node e2e test. Here is the kubelet logs portion prior to the restart: and after restart: It also looks like we are hitting another invalid state in based on the log assertion: I tried the same test on 1.26 and it passes. This is because on 1.26, the status manager phase to terminal happens at roughly at the same time when we delete the pod. On 1.27+, after , the status phase to terminal can happen earlier and the status manager can skip deleting the pod because . In this case, after start, will never be true, so the pod never will get deleted.
But in general pods can be in terminal phase and not be deleted (like plain pods and pods from jobs). They would eventually be deleted by the pod GC otherwise. Is this a bug in the device plugin?
That is true, but the issue here is that the pods have a deletion timestamp set as well. If the deletion timestamp is set and the kubelet is up, it is the kubelet's responsibility to terminate and delete those pods.

This doesn't have anything with device plugin, rather we caught the issue due to the device plugin test which asserts that all pods are terminated and deleted prior to starting the test.
I have a WIP fix for this issue in
That makes sense
Should we aim 1.27 with this fix? This seems like an important regression.
I discussed this with yesterday and we agreed that this can wait until 1.27.1 Clients should be able to handle terminated pods. Worst case scenario, the pod GC would take care of these pods eventually.
Cherrypicked to v1.27 (to be released in 1.27.2) in