Question: cmd.Flags().StringP("output", "o", "", "Output format. One of: json|yaml|template|templatefile.") cmd.Flags().String("output-version", "", "Output the formatted object with the given version (default api-version).") cmd.Flags().Bool("no-headers", false, "When using the default output, don't print headers.") <del> cmd.Flags().StringP("template", "t", "", "Template string or path to template file to use when -o=template or -o=templatefile.") </del> <ins> cmd.Flags().StringP("template", "t", "", "Template string or path to template file to use when -o=template or -o=templatefile.  The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview]") </ins> } // OutputVersion returns the preferred output version for generic content (JSON, YAML, or templates)

periodic-conformance-main-k8s-main Kubernetes e2e suite.SynchronizedBeforeSuite 07/17 23:52PDT We saw a similar issue in , which was fixed by upgrading the control plane OS from Ubuntu 18.04 to 22.04. /sig cluster-lifecycle
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
/sig node It seems to be related to SWAP.
There is a bug in containerd that is fixed in v1.6.14. The test here is using v1.6.8. Kubernetes v1.27 supported containerd versions: 1.7.0+, 1.6.15+
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still failing after is merged. Have we tried bumping the control plane OS image to ubuntu 22.04, like ?
Nevermind, the latest testgrid job might have been triggered before the PR was merged. I am observing this and will close this once I see the job passes.
Containerd version is already v1.6.21 in 's log There maybe still other problems.
can fix a similar failure in .
/close The failure of the swap is fixed by . The current failure is a flake that is mentioned in
Closing this issue. <detailsIn response to : /close The current failure is a flake that is mentioned in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: newStatus.Phase = api.PodUnknown } else { newStatus.Info = result.Status.Info <del> newStatus.Phase = getPhase(&pod.Spec, newStatus.Info) if netContainerInfo, ok := newStatus.Info[leaky.PodInfraContainerName]; ok { if netContainerInfo.PodIP != "" { newStatus.PodIP = netContainerInfo.PodIP } </del> <ins> newStatus.PodIP = result.Status.PodIP if newStatus.Info == nil { // There is a small race window that kubelet couldn't // propulated the status yet. This should go away once // we removed boundPods newStatus.Phase = api.PodPending } else { newStatus.Phase = result.Status.Phase </ins> } } return newStatus, err

Kubelet now only reports ContainerStatus. PodStatus / PodCondition is observed at master side based on a set of ContainerStatus. There is no other way to get PodIP except net container for master side. This is very confusion to the end user, , , ... Now we are finalizing on v1beta3 API, including PodStatus & ContainerStatus. We can make net container completely invisible to the end users by finishing the following steps: 1) Expose network related errors at API level 2) Generate PodStaus directly from Kubelet
3) once PodStatus generated by kubelet, PodIP can be removed from ContainerStatus.
I think reporting PodStatus rather than just container status is most important. How hard is that?
Used to be hard since there are many dependencies when I first report Status back. Now it is totally feasible. I already started to write PRs for this. :-)
One more PR required to close this one.


Question: if err == nil { t.Fatalf("MyCRD doesn't support fieldValidation, yet no error found") } <ins> err = fieldValidationVerifier.HasSupport(schema.GroupVersionKind{Group: "", Version: "v1", Kind: "List"}) if err == nil { t.Fatalf("List does not support fieldValidation, yet no error found") } </ins> }

kubectl apply performs validation before the apply. Non-CRD resources (e.g. pods / deployments) don't need CRD schema for validation, but if we use resource, kubectl calls . In my Kubernetes cluster, administrator doesn't allow users to list CRDs, so user cannot apply non-CRD resources (e.g. pods / deployments) with resource. doesn't happen because we're going to apply two Deployments. should not happen because we're going to apply two Deployments. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig cli
/triage accepted /assign
/priority backlog


Question: // ClientConfig contains all essential information to create an Azure client. type ClientConfig struct { <ins> CloudName               string </ins> Location                string SubscriptionID          string ResourceManagerEndpoint string

<!-- Please only use this template for submitting enhancement requests --What would you like to be : PR() already proves that old api-version on Azure stack works on global Azure too, we could support azure stack in azure cloud provider lib, and then make sure following components works in Azure stack: CSI drivers: azure disk, azure file CCM Since it's mainly api-version change, we could also add e2e tests against global Azure too. Why is this needed: /assign /priority important-soon /sig cloud-provider /area provider/azure
there is blocking issue here(also need azure stack supports latest data model struct): just making the change in interfaces and not go-sdk but facing this error, since we don't have a switch inside go-sdk, we only do switch in azure cloud provider lib,
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: Expression: "object.spec.replicas", }). WithVariable(admissionregistrationv1beta1.Variable{ <del> Name:       "replicasReminder", // a bit artificial but good for testing purpose Expression: "variables.replicas % 2", </del> <ins> Name:       "oddReplicas", Expression: "variables.replicas % 2 == 1", </ins> }). WithValidation(admissionregistrationv1beta1.Validation{ <del> Expression: "variables.replicas > 1 && variables.replicasReminder == 1", </del> <ins> Expression: "variables.replicas > 1", }). WithValidation(admissionregistrationv1beta1.Validation{ Expression: "variables.oddReplicas", </ins> }). Build() policy, err := client.AdmissionregistrationV1beta1().ValidatingAdmissionPolicies().Create(ctx, policy, metav1.CreateOptions{})

Variables field is to ValidatingAdmissionPolicy in 1.28. Currently the usage of variables would expect a operator associated. e.g. (thanks to Mariam Fahmy for raising the issue) Above will fail with error: Meanwhile the will pass. The reason is when variables be evaluated to non dyn type(like bool or string), it will require special handling(like casting) since we use dyn type everywhere. It might confuse user.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /sig api-machinery /cc
Tests I'd expect to see verify this behavior: PASS: A variable is of type bool, an expression returns the variable PASS: A variable is of type string, a messageExpression returns the variable PASS: A variable is of type dyn, at runtime the dyn is a bool, an expression returns the variable PASS: A variable is of typedyn, at runtime the dyn is a string, an messageExpression returns the variable PASS: A expression is of type dyn, at runtime the dyn is of type bool PASS: A messageExpression is of type dyn, at runtime the dyn is of type string COMPILE ERROR: A variable is of type bool, an expression tries to perform a non-bool (e.g. string replace) operation on the variable, this should resultsin a compile error COMPILE ERROR: A variable is of type string, an expression tries to perform a non-string (e.g. less than) operation on the variable, this should result in a compile error RUNTIME ERROR: A variable is of type dyn, at runtime the dyn is a bool, an expression tries to perform a non-bool (e.g. string replace) operation on the variable, this should result in a runtime error I'd also expect to see some cases covering variables of other types (maps, lists, duration, quantity, ...) to make sure we're over-fitting for bool and string.
/retitle ValidatingAdmissionPolicy: variable expression type not recognized as return type in expression that evaluates the variable
/assign


Question: s.readCalled = true b := []byte(s.data) n := copy(p, b) <ins> // Indicate we returned all the data, and have no more data (EOF) // Returning an EOF here will cause the port forwarder to immediately terminate, which is correct when we have no more data to send </ins> return n, io.EOF }

ok 1.520s coverage: 25.5% of statements I0414 23:19:35. ] Forwarding from 5000 -5000 W0414 23:19:35. ] Unable to listen on port {5000 5000}: listen tcp 127.0.0.1:5000: bind: address already in use I0414 23:19:35. ] Forwarding from 6000 -6000 I0414 23:19:35. ] Handling connection for 5000 --- FAIL: TestForwardPorts (0.22s) 2: expected to read '1234', got '' FAIL coverage: 83.3% of statements FAIL 0.677s
It looks like the crux is this: ] Unable to listen on port {5000 5000}: listen tcp 127.0.0.1:5000: bind: address already in use listen often needs a retry after a recent close, to account for TIME_WAIT state of the port. But that's pure speculation on my part. We need to repro and fix.
http://hea- is perhaps a better link.
I'll have a go at it in the next few days if nobody else picks it up before me.
There were two issues... one was multiple test cases reusing the same port and deferring the close until the end of the function, not the end of the loop... that was fixed in The other issue was with the fake stream returning an EOF on a Write(), fixed in


Question: // +optional ResourceVersion string `json:"resourceVersion,omitempty" protobuf:"bytes,4,opt,name=resourceVersion"` // Timeout for the list/watch call. <ins> // This limits the duration of the call, regardless of any activity or inactivity. </ins> // +optional TimeoutSeconds *int64 `json:"timeoutSeconds,omitempty" protobuf:"varint,5,opt,name=timeoutSeconds"`

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Watches done with the field of set to some value T end after T regardless of whether there has been any activity in the meantime. What you expected to happen: I expected the watch to end when no events have been delivered in the last T. In ordinary usage, a "timeout" is something that happens if there has been no activity for the specified length of time. How to reproduce it (as minimally and precisely as possible): Create a configmap named in namespace . Then in one shell thrash it periodically like this: Observe that it proceeds to update the configmap every 7 seconds or so. In another shell do a watch with a timeout; observe that it ends even though the thrashing continues. Anything else we need to know?: Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.4+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-21T17:22:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}Cloud provider or hardware configuration: local hardware OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): Linux n17s2ctrl 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: local Others:
I am told the observed behavior is the intended behavior, the problem here is that the field name and comment are misleading.
/sig api-machinery
It's a timeout on the request lifetime, not an inactivity timeout
The inactivity condition is implied in ordinary usage of the word "timeout". A word like "lifetime", "duration", or "time limit" would be a better description of the existing behavior. I do not propose changing the field name for this, that would be more work than it is worth. I propose updating the commentary on that field to make its meaning clear. BTW, what exactly is this field's meaning to a list operation?


Question: spec: containers: - name: kibana-logging <del> image: docker.elastic.co/kibana/kibana-oss:7.1.1 </del> <ins> image: docker.elastic.co/kibana/kibana-oss:7.2.0 </ins> resources: # need more cpu upon initialization, therefore burstable class limits:

What happened: On latest fluentd-elasticsearch, the elasticsearch replicas seems failed to form a valid cluster with the service discovery helper written in golang. What you expected to happen: Successfully boot up an elasticsearch cluster I guess. How to reproduce it (as minimally and precisely as possible): Just use the yamls in and the elasticsearch-logging pods will keep spamming warning messages. I think thats because options are deprecated in elasticsearch 7.x.x Anything else we need to know?: IMHO I think thats all. Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig instrumentation
Thanks


Question: "io/ioutil" "net/http" "net/http/httptest" <ins> "os" </ins> "reflect" "runtime" "sync"

E1003 10:53:11. ] Could not read directory: , (open : no such file or directory) This happens many times


Question: defer cleanupNetworkPolicy(f, policy) ginkgo.By(fmt.Sprintf("Creating client pod %s that should not be able to connect to %s.", "client-a", service.Name)) <del> podClient := createNetworkClientPod(f, f.Namespace, "client-a", service, allowedPort) </del> <ins> // Specify RestartPolicy to OnFailure so we can check the client pod fails in the beginning and succeeds // after updating its label, otherwise it would not restart after the first failure. podClient := createNetworkClientPodWithRestartPolicy(f, f.Namespace, "client-a", service, allowedPort, v1.RestartPolicyOnFailure) </ins> defer func() { ginkgo.By(fmt.Sprintf("Cleaning up the pod %s", podClient.Name)) if err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Delete(context.TODO(), podClient.Name, nil); err != nil { framework.Failf("unable to cleanup pod %v: %v", podClient.Name, err) } }() <del> checkNoConnectivity(f, f.Namespace, podClient, service) </del> <ins> // Check Container exit code as restartable Pod's Phase will be Running even when container fails. checkNoConnectivityByExitCode(f, f.Namespace, podClient, service) </ins> ginkgo.By(fmt.Sprintf("Updating client pod %s that should successfully connect to %s.", podClient.Name, service.Name)) <del> podClient = updateNetworkClientPodLabel(f, f.Namespace, podClient.Name, "replace", "/metadata/labels", map[string]string{}) </del> <ins> podClient = updatePodLabel(f, f.Namespace, podClient.Name, "replace", "/metadata/labels", map[string]string{}) </ins> checkConnectivity(f, f.Namespace, podClient, service) }) <ins> ginkgo.It("should deny ingress access to updated pod [Feature:NetworkPolicy]", func() { const allowedPort = 80 ginkgo.By("Creating a network policy for the server which denies all traffic.") policy := &networkingv1.NetworkPolicy{ ObjectMeta: metav1.ObjectMeta{ Name: "deny-ingress-via-isolated-label-selector", }, Spec: networkingv1.NetworkPolicySpec{ PodSelector: metav1.LabelSelector{ MatchLabels: map[string]string{ "pod-name": podServerLabelSelector, }, MatchExpressions: []metav1.LabelSelectorRequirement{{ Key:      "isolated", Operator: metav1.LabelSelectorOpExists, }}, }, PolicyTypes: []networkingv1.PolicyType{networkingv1.PolicyTypeIngress}, Ingress:     []networkingv1.NetworkPolicyIngressRule{}, }, } policy, err := f.ClientSet.NetworkingV1().NetworkPolicies(f.Namespace.Name).Create(context.TODO(), policy, metav1.CreateOptions{}) framework.ExpectNoError(err, "Error creating Network Policy %v: %v", policy.ObjectMeta.Name, err) defer cleanupNetworkPolicy(f, policy) // Client can connect to service when the network policy doesn't apply to the server pod. testCanConnect(f, f.Namespace, "client-a", service, allowedPort) // Client cannot connect to service after updating the server pod's labels to match the network policy's selector. ginkgo.By(fmt.Sprintf("Updating server pod %s to be selected by network policy %s.", podServer.Name, policy.Name)) updatePodLabel(f, f.Namespace, podServer.Name, "add", "/metadata/labels/isolated", nil) testCannotConnect(f, f.Namespace, "client-a", service, allowedPort) }) </ins> ginkgo.It("should enforce egress policy allowing traffic to a server in a different namespace based on PodSelector and NamespaceSelector [Feature:NetworkPolicy]", func() { var nsBserviceA, nsBserviceB *v1.Service var nsBpodServerA, nsBpodServerB *v1.Pod

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The NetworkPolicy e2e test fails regardless of which CNI plugin is enabled. The test assumes the client Pod can recheck connectivity after updating its label, but the client won't restart after the first failure, so the second check will always fail. What you expected to happen: The test should succeed if a CNI plugin implements NetworkPolicy correctly. How to reproduce it (as minimally and precisely as possible): a CNI that supports NetworkPolicy. the e2e test by focusing "should allow ingress access from updated pod". test will always fail. Anything else we need to know?: Environment: Kubernetes version (use ): 16.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing /assign
/remove-sig testing /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I see this in Calico 3.10 as well confirming that its definitely an issue.


Question: if o.Attach { if remove { <del> defer o.removeCreatedObjects(f, createdObjects) </del> <ins> defer func() { if err := o.removeCreatedObjects(f, createdObjects); err != nil { fmt.Fprintf(o.ErrOut, "Delete failed: %vn", err) } }() </ins> } opts := &attach.AttachOptions{

The command provides a flag but this doesn't do anything. most likely the flag shouldn't exist. I have no idea how kustomize could be useful here to start a single pod. This creates a pod, and shows that is ignored: No response <details</details<detailsAzure, K3OS</details<detailsOSX 11.6.1 (Big Sur) </detailsNo response No response No response
/sig cli
This comes from the inclusion of DeleteFlags: brings in FilenameFlags: has Kustomize, resulting in the flag: I suspect there are many flags from DeleteFlags that don't really apply to . : Not sure what this would do : seems to be ignored and : doesn't seem to be used : doesn't seem to be used : also probably doesn't apply There might be others as well...
It seems to delete object which created by command.
I agree, As far as I can see, , and have this deleteFlags exposure.
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: return "", false, err } for _, name := range pods { <del> if strings.Contains(name, networkContainerName+"--"+manifest.Id+"--") { </del> <ins> if strings.Contains(name, containerNamePrefix+"--"+networkContainerName+"--"+manifest.Id+"--") { </ins> return name, true, nil } }

release-master-blocking Conformance-GCE-master-kubetest2Kubernetes e2e suite.[It] [sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance] ci-kubernetes-gce-conformance-latest-kubetest2.Overall kubetest2.Test 07-11 15:51 EEST No response /sig cloud-provider cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I remember some timeout has changed with ginkgo2 and something moved from 24hours to 1 hours, or something like that, is it possible that this is the problem here, in this job?
The Grinko timeout change could also be the root problem with this job:
hmm, we have to revisit the flags
hmm, this seems correct
that was a good observations, seems that job is failing because gathering the logs takes too much time !
there is no timeout set in the kubetest2 command here (not sure if this would be the place to set a timeout)
it seems kubetest has to be migrated to ginkgov2 it should use this as basis I'm on vacation this week, can somebody take over and open the issue and patch kubetest2?
looks like this file needs to get updated: add line 72:
oh, you already commented this :D
I will have a look later
Related: umbrella issue:
last testgrid run passed :)


Question: "description": "CustomResourceConversion describes how to convert different versions of a CR.", "properties": { "strategy": { <del> "description": "strategy specifies how custom resources are converted between versions. Allowed values are: - `None`: The converter only change the apiVersion and would not touch any other field in the custom resource. - `Webhook`: API Server will call to an external webhook to do the conversion. Additional informationn  is needed for this option. This requires spec.preserveUnknownFields to be false, and spec.conversion.webhook to be set.", </del> <ins> "description": "strategy specifies how custom resources are converted between versions. Allowed values are: - `"None"`: The converter only change the apiVersion and would not touch any other field in the custom resource. - `"Webhook"`: API Server will call to an external webhook to do the conversion. Additional informationn  is needed for this option. This requires spec.preserveUnknownFields to be false, and spec.conversion.webhook to be set.", </ins> "type": "string" }, "webhook": { "$ref": "#/definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.WebhookConversion", <del> "description": "webhook describes how to call the conversion webhook. Required when `strategy` is set to `Webhook`." </del> <ins> "description": "webhook describes how to call the conversion webhook. Required when `strategy` is set to `"Webhook"`." </ins> } }, "required": [

Currently we have inconsistency in most of the API where the field name should be used in description so the generated docs match the API fields people actually set. A linter could be for checking To avoid confusing users
/good-first-issue
/triage accepted
/assign
Hey can you provide some more information for this issue
are you still working on this?
Hey feel free to take it. I am unassigning myself
/triage needs-information /remove-good-first-issue
/sig api-machinery


Question: srcs = ["testfiles.go"], importpath = "k8s.io/kubernetes/test/e2e/framework/testfiles", visibility = ["//visibility:public"], <del> deps = ["//vendor/github.com/onsi/ginkgo:go_default_library"], </del> <ins> deps = ["//test/e2e/framework/log:go_default_library"], </ins> ) filegroup(

/help /kind cleanup /area e2e-testing-framework mini guide: make sure you have signed the CNCF clone kubernetes;/kubernetes search for ginkgo.Fail calls under test/e2e/framework e.g. replace them with this that was in this PR: send a PR for kubernetes/kubernetes
/remove-kind feature /area e2e-test-framework
/sig testing
I will take this.
thank you! Please feel free to ping #testing-commons in the Kubernetes slack if you have any questions :smiley: /cc
/remove-help
/assign


Question: } } <ins> // runStaticPodTest is disabled until #6651 is resolved. </ins> func runStaticPodTest(c *client.Client, configFilePath string) { var testCases = []struct { desc         string

Probably not the same as ? To for triage
Shouldn't this fixed by ?
this has nothing to do with . kubelet is trying to create the mirror pod over and over again because it still hasn't observed the mirror pod from watch. It is probably the same as # .
I thought they might be different because this one has a bunch of "failed creating ... already exists" messages in it, but doesn't.
It is essentially the same problem -- the watch was delayed so kubelet kept seeing the old snapshot of the pods where the mirror didn't exist. That's why it's flaky and only happens on shippable (which I suspect has poorer performance) :( This becomes worse every time a new test is . I think we need to limit the number of concurrent tests in
limits the number of concurrent tests on travis/shippable to 4. Let's see if this helps reduce the timeout flakes.
Another example:
Here is yet another example: Retitled the issue since limiting the concurrent tests doesn't work. It could be that the server is heavily loaded and the test timed out, or there was a delay/drop in kubelet's pod watch stream.
Now example:
If, for whatever reason, a watch update for pod creation is dropped, kubelet would need to wait until re-listing to notice that. This would certainly cause the test to time out. would it be useful to shorten the re-listing interval for tests like this?
I am not quite sure how this works--let's chat about it tomorrow?
From , we can see that kubelet was lightly loaded as the update channel was mostly empty while waiting. This corroborates the theory that watch might be at fault.
cc: quinton-hoole
The WIP reflector benchmark () may shed some light on this issue once it's integrated with the continuous e2e performance suite. I will keep this issue open for now to gather more data (since this is not reproducible locally at all), and to see if there are other potential causes.
The PR above disables the static pod test temporarily so that we can track down the root cause without spamming people with the failures. Note that other tests in the same file () may still trigger the flaky failure, though less frequently.
Can this be closed yet?
No, integration test is still flaky. See for example.

Many tests in rely on checking the pod status with a preset timeout. I disabled one test which failed most often, but other tests still fail from time to time.
The issue has been inactive for months, and there have not been any new reports. Closing.


Question: This time the label query for the service is `name=redis-slave`. <del> Now that you have created the service specification, create it in your cluster with the `cloudcfg` cli: </del> <ins> Now that you have created the service specification, create it in your cluster with the `cloudcfg` CLI: ```js $ cluster/cloudcfg.sh -c examples/guestbook/redis-slave-service.json create /services Using master: kubernetes-master (external IP: 1.2.3.4) { "id": "redisslave", "port": 10001, "labels": { "name": "redisslave" } } </ins> <del> ```shell cluster/cloudcfg.sh -c redis-slave-service.json create /services </del> ``` ### Step Five: Create the frontend pod. <del> This is a simple PHP server that is configured to talk to both the slave and master services depdending on if the request is a read or a write.  It exposes a simple AJAX interface, and serves an angular based U/X.  Like the redis read slaves it is a replicated service instantiated by a replication controller.  Create a file named `frontend-controller.json`: </del> <ins> This is a simple PHP server that is configured to talk to either the slave or master services depending on whether the request is a read or a write. It exposes a simple AJAX interface, and serves an angular-based UX. Like the redis read slaves it is a replicated service instantiated by a replication controller. </ins> <del> ```javascript { "id": "frontendController", "desiredState": { "replicas": 3, "replicasInSet": {"name": "frontend"}, "podTemplate": { "desiredState": { "manifest": { "containers": [{ "image": "brendanburns/php-redis", "ports": [{"containerPort": 80, "hostPort": 8080}] }] } }, "labels": {"name": "frontend"} }}, "labels": {"name": "frontend"} } </del> <ins> Create a file named `frontend-controller.json`: ```js { "id": "frontendController", "desiredState": { "replicas": 3, "replicasInSet": {"name": "frontend"}, "podTemplate": { "desiredState": { "manifest": { "containers": [{ "image": "brendanburns/php-redis", "ports": [{"containerPort": 80, "hostPort": 8080}] }] } }, "labels": {"name": "frontend"} }}, "labels": {"name": "frontend"} } </ins> ``` With this file, you can turn up your frontend with: <del> ```shell cluster/cloudcfg.sh -c frontend-controller.json create /replicationControllers </del> <ins> ```js $ cluster/cloudcfg.sh -c examples/guestbook/frontend-controller.json create /replicationControllers Using master: kubernetes-master (external IP: 1.2.3.4) { "id": "frontendController", "desiredState": { "replicas": 3, "replicasInSet": { "name": "frontend" }, "podTemplate": { "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/php-redis", "ports": [ { "hostPort": 8080, "containerPort": 80 } ] } ] } }, "labels": { "name": "frontend" } } }, "labels": { "name": "frontend" } } </ins> ``` Once that's up you can list the pods in the cluster, to verify that the master, slaves and frontends are running: <del> ```shell cluster/cloudcfg.sh list /pods </del> <ins> ```js $ cluster/cloudcfg.sh list /pods Using master: kubernetes-master (external IP: 1.2.3.4) { "kind": "cluster#podList", "items": [ { "id": "redis-master-2", "labels": { "name": "redis-master" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "name": "master", "image": "dockerfile/redis", "ports": [ { "hostPort": 6379, "containerPort": 6379 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-3.c.{project-id}.internal" } }, { "id": "4d65822107fcfd52", "labels": { "name": "redisslave", "replicationController": "redisSlaveController" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/redis-slave", "ports": [ { "hostPort": 6380, "containerPort": 6379 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-3.c.{project-id}.internal" } }, { "id": "55104dc76695721d", "labels": { "name": "frontend", "replicationController": "frontendController" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/php-redis", "ports": [ { "hostPort": 8080, "containerPort": 80 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-3.c.{project-id}.internal" } }, { "id": "78629a0f5f3f164f", "labels": { "name": "redisslave", "replicationController": "redisSlaveController" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/redis-slave", "ports": [ { "hostPort": 6380, "containerPort": 6379 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-2.c.{project-id}.internal" } }, { "id": "380704bb7b4d7c03", "labels": { "name": "frontend", "replicationController": "frontendController" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/php-redis", "ports": [ { "hostPort": 8080, "containerPort": 80 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-2.c.{project-id}.internal" } }, { "id": "365a858149c6e2d1", "labels": { "name": "frontend", "replicationController": "frontendController" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/php-redis", "ports": [ { "hostPort": 8080, "containerPort": 80 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-4.c.{project-id}.internal" } } ] } </ins> ``` <del> You should see a single redis master pod, two redis slave and three frontend pods. </del> <ins> You will see a single redis master pod, two redis slaves, and three frontend pods. </ins> The code for the PHP service looks like this: ```php <?

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am running an admission webhook using If there it has been hours since the last mutating request to the webhook, then the next mutating request will take many seconds to be processed. This may even take long enough to have the request time out. What you expected to happen: A request takes approximately the same amount of time to be processed regardless of how long it has been since the last request. How to reproduce it (as minimally and precisely as possible): Create an API server for handling a new API group. Wait a couple hours. Send a CREATE request to the API server. Observe that the CREATE request takes a few seconds to complete. Anything else we need to know?: This is the offending code. It must loop through once for every second since the last request. Environment: Kubernetes version (use ): 1.19.0
/sig api-machinery
I thought we'd already merged a fix for this from before the release?
I assume that the fix was supposed to be From my understanding, those changes only are in effect if the apiserver is running with priority and fairness enabled. In my situation, I do not have that enabled.
Ack, if we have the problem in both places we need the fix in both places :)
/triage accepted
I think we have fixed this in , do you guys agree?


Question: // UnmountDevice unmounts the volume on the node func (d *azureDiskDetacher) UnmountDevice(deviceMountPath string) error { <ins> if runtime.GOOS == "windows" { // Flush data cache for windows because it does not do so automatically during unmount device exec := d.plugin.host.GetExec(d.plugin.GetPluginName()) err := util.WriteVolumeCache(deviceMountPath, exec) if err != nil { return err } } </ins> err := mount.CleanupMountPoint(deviceMountPath, d.plugin.host.GetMounter(d.plugin.GetPluginName()), false) if err == nil { klog.V(2).Infof("azureDisk - Device %s was unmounted", deviceMountPath)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-azure-disk-windows Which test(s) are flaking: [Fail] Dynamic Provisioning [single-az] [It] should create a statefulset object, write and read to it, delete the pod and write and read to it again [] [] [Windows] Testgrid link: Reason for failure: Unknown Anything else we need to know: Unexpected error: <*errors.errorString | 0xc0007c26a0: { s: "Failed to find "hello worldrnhello worldrn", last result: "x00x00x00x00x00x00x00x00x00x00x00x00x00hello worldrn"", } Failed to find "hello world hello world ", last result: " hello world " occurred
/sig cloud-provider /area provider/azure
Ref: I will try to solve it.
Ref:
/assign
The PR kubernetes-sigs/azuredisk-csi-driver should fix the error. Let's observe this test case for a period of time. When it is stable, could close the issue.


Question: QOSReserved: {Default: false, PreRelease: featuregate.Alpha}, <del> ReadWriteOncePod: {Default: false, PreRelease: featuregate.Alpha}, </del> <ins> ReadWriteOncePod: {Default: true, PreRelease: featuregate.Beta}, </ins> RecoverVolumeExpansionFailure: {Default: false, PreRelease: featuregate.Alpha},

For 1.27 we should graduate the ReadWriteOncePod feature to beta. This is needed to rollout the feature to GA. In beta this feature will support preemption of pods using PVCs. See:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /sig scheduling


Question: package container import ( <ins> "encoding/json" </ins> "fmt" "hash/fnv" "strings"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: During the update from v1.7.1 to v1.8.0 all docker containers were restarted, because container spec hash changed. What you expected to happen: No restart of the docker containers, because they did not change at all. How to reproduce it (as minimally and precisely as possible): get yourself a k8s cluster in v1.7.x update (at least the kubelet) to v1.8.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/area kubelet
I ran into this earlier today. I was able to update the control plane & kube-proxy to 1.8.1 without any problems. Once I upgraded the kubelet though it caused container restarts.
/sig apps /sig api-machinery
the log message one receives is at: The function, that calculates the hashes is at: The hash function seems to not have changed. I suspect changes in the object itself, which is at: One can see: but:
But the new field does not initiated, so the restart seems unreasonable.
the result i tested was that the spew pkg also dumps the new add field(with nil value) to the hasher. is the new field. so, if the container use , then new MountPropagationMode will be included into the hash input, the hash result changed. other new fields are in the same situation.
Do you have any idea about preventing this?
We've hit this as well. Caused a cluster-wide restart of every container.
i also hit this issue as well, when I am upgrading cluster from v1.7.4 to v1.8.5. especially, the kubelet version from v1.7.4 to v1.8.5
Still seems to happen in 1.9.0
I can confirm this. v1.9.0 has introduced a new field attribute for , which is causing the hash value get changed.
It will exist when we use docker. I remember had described it.

There are details in this issues below:
thanks for digging out the old issue! Yes, any new field to the container spec would cause the hash to change. This is not a new issue but new fields are to the container spec rarely (compared with pod spec) so it doesn't happen so often. As discussed in , a new way to generate the hash (e.g., using only updatable fields) or other method for kubelet to examine and compare the state of the container against the spec is needed. Calling this (i.e., container restart) out in the 1.9 release note would be nice.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/close
Just wondering, why don't we calculate the container hash by since container image is the only mutable field? I know we have to do some refactoring to , but wouldn't it be more reliable?


Question: return fmt.Errorf("invalid disk to detach: %q", diskURI) } <del> _, err := d.cloud.InstanceID(context.TODO(), nodeName) </del> <ins> instanceid, err := d.cloud.InstanceID(context.TODO(), nodeName) </ins> if err != nil { glog.Warningf("no instance id for node %q, skip detaching (%v)", nodeName, err) return nil

/kind bug What happened: I'm running a POD with three VolumeMount (AzureDisk PVC). If I delete the pod, it gets rescheduled on a different node. The 3 disks are unmounted from the original node (thanks to kubelet) and the disks are starting to get unattached from the VM. Out of the three disks, only one was able to get unattached, it was properly reattached to the new node waiting to run the freshly restarted pod. The other two Disks couldn't be detached. The controller manager keeps trying to attach from this point on while never trying to re-detach. What you expected to happen: Kube-controller-manager should retry detaching disks after failures. How to reproduce it (as minimally and precisely as possible): create a pod with 3 volume mounts (AzureDisk). Kill the running pod. watch the status If it works, retry Anything else we need to know?: kube-controller-manager logs (--v=4) Environment: Kubernetes version (use ): 1.7.12 Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): centOS 7.4 Kernel (e.g. ): 3.10.0-693.11.6.el7.x86_64 Install tools: in-house (terraform + salt) cc:
/sig azure /sig storage
/assign let me know if you need more logs.
Three disk ID. one was detached / reattached properly the other twos had errors since they were detached at the same time (3 concurrent detach, 1 pass, 2 fails) After that the remaining two start looping with
This is one problem. the disk is still attached to the VM.
Azure Portal shows that a bunch of errors occurred, followed by three success. Each of those should have been the removal of a single disk. In the end, only 1 disk disappeared. <img width="508" alt="screen shot 2018-02-21 at 01 21 23" src="https://user-"<img width="591" alt="screen shot 2018-02-21 at 01 21 32" src="https://user-"My gut feeling is that the backoff logic doesn't refresh the list of disks to apply on the VM.

So we have three functions running all trying to delete a single disk. The deletion is performed using a VMUpdate call with a list of Disks (minus the one we want to remove) The new VM is construction based on the list of disks (-1) All three deletion logics start at the same time, causing race conditions with the azure api. or Eventually, the backoff logic will get through and the last VM update will re-add the two disks that were not part of the original deletion request. One solution could be to lock on disk deletion if they touch the same NodeName? Or queue the deletion request when an existing request is ongoing for the same node.
you are right, there is no locking for detach disk, I will try to repro it and then use the locking fix to verify it, thanks for reporting.
/assign
let me know if you can’t reproduce. It’s easy for me. I could cook you up a quick manifest. Thanks for looking into it.
Your repro shows obvious evidence of race condition when detaching azure disk, I have commited a PR, pls take a look. Thanks for your reporting, it's really helpful.


Question: c = f.ClientSet }) <del> ginkgo.It("should create pod that uses dns", func() { mkpath := func(file string) string { return filepath.Join(os.Getenv("GOPATH"), "src/k8s.io/examples/staging/cluster-dns", file) </del> <ins> read := func(file string) string { data, err := e2etestfiles.Read(file) if err != nil { framework.Fail(err.Error()) </ins> } <ins> return string(data) } ginkgo.It("should create pod that uses dns", func() { </ins> // contrary to the example, this test does not use contexts, for simplicity // namespaces are passed directly. // Also, for simplicity, we don't use yamls with namespaces, but we // create testing namespaces instead. <del> backendRcYaml := mkpath("dns-backend-rc.yaml") backendRcName := "dns-backend" backendSvcYaml := mkpath("dns-backend-service.yaml") backendSvcName := "dns-backend" backendPodName := "dns-backend" frontendPodYaml := mkpath("dns-frontend-pod.yaml") frontendPodName := "dns-frontend" frontendPodContainerName := "dns-frontend" </del> <ins> backendName := "dns-backend" frontendName := "dns-frontend" clusterDnsPath := "test/e2e/testing-manifests/cluster-dns" </ins> podOutput := "Hello World!" // we need two namespaces anyway, so let's forget about

What happened: We've been doing some release readiness and auditing all tagged ginkgo tests... Looks like The test seems to expect a GOPATH when running. Im seeing some failures in sonobuoy and suspect a possible relation there but havent dove into it. Either way, its clear this test might need some updating/fixing/cleanuping and parts can be rewritten/improved. What you expected to happen: No reliance on GOPATH for an e2e test ... I dont remember the whole history on this, but no tests that rely on static files currently look in gopath, from what i can tell ? instead we plumbed in a repo-root/ option a few years ago. There may be a state of the art in terms of how we do this.... but i thought it was at one point which was an option passed in. How to reproduce it (as minimally and precisely as possible): Try to run this in sonobuoy or any other environment where there's no GOPATH or non-standard source trees, pretty sure it might not work. Anything else we need to know?: I think the fix here is to just embed the YAML in golang like we do in other tests. along the way there's other cleanup we can do to improve this test. Environment: Kubernetes version (use ): master
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
the YAML files are pretty small, so +1 to embed them in i don't think testing the YAML from the separate repository () directly is required here. ref /help /kind cleanup /priority backlog
/remove-help
/assign Also I should say, I talked to Jay on slack and he is ok with me taking a look at it, if thats ok.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle stale
/remove-lifecycle stale
xref: go 1.16 has built in support for embedding go 1.17 looks to be removing GOPATH
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/help wanted
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten still a problem. FYI re: GOPATH removal
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/assign


Question: } e.recordToSink(event, clock.RealClock{}) } <del> stopWatcher := e.StartEventWatcher(eventHandler) </del> <ins> stopWatcher, err := e.StartEventWatcher(eventHandler) if err != nil { return err } </ins> go func() { <-stopCh stopWatcher() }() <ins> return nil </ins> } // StartRecordingToSink starts sending events received from the specified eventBroadcaster to the given sink. func (e *eventBroadcasterImpl) StartRecordingToSink(stopCh <-chan struct{}) { go wait.Until(e.refreshExistingEventSeries, refreshTime, stopCh) go wait.Until(e.finishSeries, finishTime, stopCh) <del> e.startRecordingEvents(stopCh) </del> <ins> err := e.startRecordingEvents(stopCh) if err != nil { klog.Errorf("unexpected type, expected eventsv1.Event") return } </ins> } type eventBroadcasterAdapterImpl struct {

Follow-up cleanup work for: Most of the information about this issue can be found in , where the work merged with a to change the function signature of to return a func() as well as an error. This issue will help to track that. All TODOs introduced in are resolved. N/A N/A <detailsN/A </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority low Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /triage accepted
/priority important-longterm
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: utilsexec "k8s.io/utils/exec" utilsnet "k8s.io/utils/net" <del> "github.com/PuerkitoBio/purell" </del> "github.com/pkg/errors" )

This is a proactive issue opened to explore the path forward with the module that we have been using as a dependency for in here This module is looking for a new maintainer. Discussions around the path forward, potentially eliminate the module from usage and do this operation in-house, as we use it in only one place for specific functionality. Ref Issue from repo: Slack Thread:
in kubeadm the usage of : Unicode normalization on the URL host according to "Unicode Normalization Form C". Unicode transformation to "fold" characters to canonical width on the host. the host from Unicode to ASCII (IDNA encoding, but 1 and 2 and needed due to a non-conformant x/net/idna at the time?) duplicate "//" from the URL path. the whole URL as per RFC 3986 (looks like PuerkitoBio/urlesc forks golang/net/url). IMO, we should: replace the usage of with this trimmed local version: then add a release note to the PR: we could import , and do the punycode conversion, but i do not understand why it should be kubeadm's responsibility to do that. it feels like IDNA is a host reachability problem that should be solved pre-k8s deployment. i can see this being a breaking change to users having Unicode etcd endpoints in kubeadm, but i'm hoping the same users were already aware to not rely on k8s components to manage this for them.
/sig cluster-lifecycle /area kubeadm /remove-kind bug /kind cleanup
cool, that was helpful. I'll take up on this and create a PR.
/assign
From a Cluster API perspective: It's highly unlikely from the CAPI side that we'll specify IDNA in the kubeadm config. There is a chance on CAPV (vSphere) that a host picks up the DHCP option for the domain name suffix, and that it's using an IDNA. E.g., in my home environment, i have DHCP option 15 set to , and these end up in the etcd pods as parameters, e.g. Is IDNA encoding a consideration for input in etcd's command line, or are unicode args fine?

For the public clouds, it's even more unlikely we'll hit it as the choice of domain name suffix is highly constricted to meet node identity requirements in the CPI. My own take, is that I'm not wed to keeping this around at all, I haven't seen a use case for it.
cc and for any further vsphere comments.
have you seen DHCP with IDNA domains used for external etcd with kubeadm?
unicode args should be fine, also apparently Go's HTTP stack does IDNA encoding: i will update the proposed release note above. but IMO, if one would like a domain to be encoded when passed to all the tools in the k8s stack, maybe it should be done in advance. it feels like IDNA is a host reachability problem that should be solved pre-k8s deployment. or leave it to the Go standard library to do the encoding. i saw comments that it might not be fully compliant yet, but i don't know how up to date these comments are.
/triage accepted i didn't see any PRs for this for 1.22. looks like we can update it in 1.23.
Apologies, I was a little occupied at work. But I have created the PR Please let me know if we can add this as part of 1.22.
no worries. we cannot add it in 1.22 because it's not a critical bug fix and we are in code freeze. i will comment on the PR.
looks like we should tell go-openapi/jsonreference to stop using the library too.
done. Created an issue in their repo as well and offered to create a PR if they are willing to move forward with in-sourcing the logic.
thanks!
just an FYI, the maintainers of the go-openapi have also removed the purell dependency.


Question: t.Run(test.name, func(t *testing.T) { serviceName := types.NamespacedName{Namespace: "default", Name: "myservice"} <del> sgList, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) </del> <ins> sgList, setupSg, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) </ins> assert.NoError(t, err, "buildELBSecurityGroupList failed") extraSGs := sgList[1:] assert.True(t, sets.NewString(test.expectedSGs...).Equal(sets.NewString(extraSGs...)), "Security Groups expected=%q , returned=%q", test.expectedSGs, extraSGs) <ins> assert.True(t, setupSg, "Security Groups Setup Permissions Flag expected=%t , returned=%t", true, setupSg) </ins> }) } }

What happened: Create SG outside of K8s - a SG rule has a range of ports which are opened via K8s Add this into Ensure load balancer switches from the range back to individual ports, the CIDR will be 0.0.0.0/0 if loadBalancerSourceRanges was removed too What you expected to happen: K8S doesn't modify the SG How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.7 Cloud provider or hardware configuration: EKS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig aws
Note - I'm not sure if this is related to the aws cloud config, which I can't change as I'm using EKS Add aws cloud config: [global] disableSecurityGroupIngress = true
/sig cloud-provider
We're also having this issue which makes kinda unusable. shouldn't ensure permissions if an SG is a custom one here . could you please take a look?
This is also an issue for us, using these service annotations should have been straightforward. A custom security group id used in the should ensure it's attached to the ELB/LB, and remain untouched. Instead, this customer security group gets purged, leaving the port mapping untouched, but all sources become 0.0.0.0/0.
unfortunately I can't, I don't have access at AWS resources to work on that. To be honest, it was pointed out a number of times that the load balancer code will need a refactor and better tests, I would suggest pinging AWS directly or bringing it up during the meetings.
Suppose you have an existing EKS cluster, gateways, the whole nine yards... Also, suppose you have a custom security group with entries already defined, for example, an SG that holds the rules of your corporate office network ranges, or even customer-specific IPs. You expect, by using this SG id (sg-) in conjunction with this service annotation that: All other security groups kubernetes created during init would be "wiped away", and your custom SG would be the load-balancer's only security group (assuming you only had one).

All entries in your custom security group are intact, and would be retained by the control plane. Gotcha, looks like it doesn't. It's reverted back to 0.0.0.0/0 as the source for all rules. Temporary solution: Run the override script (in our case, assign the serviceAnnotations to the gateway). Still use a custom security group, but leave it empty. When all and done, add your custom entries to the SG. As of 08/22, this cheap workaround, works. Still needs investigating.
Any subsequent updates to the cluster, reverts all configurations back to 0.0.0.0/0. EKS was updated from eks.2 to eks.3. Which somehow triggered the changes with istio.
Hi, It looks I have the same issue. Company policy dictates that all SG's have a specific name. Before: we just had service deployment which result in an auto-generated ELB with an auto-generated SG with the 'wrong' name, but everything was working fine. I'm now in the process of getting the correct name. So I created a custom SG "A" which has ingress rules SG "A1" and "A2". I add this with Now the elb is still auto-generated (as expected), but it only has 1 of the SG's from "A", being "A2" and a autogenerated one. I the a file with contents in the location. But I cannot get it to work. We use as , which probably creates this extra rule. I cannot find ( or am probably not able to read correctly) the correct config. Should the aws-load-balancer-internal be used or not? Do I need another directive? Maybe someone can help me out with some pointers? Thanks in advance.
It's possible to work around this by setting to something bogus (i.e. ) and putting your desired SG in the annotation instead (k8s won't modify this one).
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: func (fv *FakeVolume) Detach(volumeName string, nodeName types.NodeName) error { fv.Lock() defer fv.Unlock() <del> fv.DetachCallCount++ </del> node := string(nodeName) volumeNodes, exist := fv.VolumesAttached[volumeName]

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-unit : links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
still flaking /remove-lifecycle stale
again
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: mkdir -p /etc/salt/minion.d cat <<EOF >/etc/salt/minion.d/master.conf master: '$(echo "$MASTER_NAME" | sed -e "s/'/''/g")' <ins> master: '$(echo "$MASTER_NAME" | sed -e "s/'/''/g")' auth_timeout: 10 auth_tries: 2 auth_safemode: True ping_interval: 1 random_reauth_delay: 3 state_aggregrate: - pkg </ins> EOF cat <<EOF >/etc/salt/minion.d/grains.conf

It appears that the provision-network script that is run on the kubernetes-master can cause salt provisioning on the minions to fail or block in a forever retry timeout error loop. I think it's because of the service network.restart that happens after openvswitch is installed in provision-network. I only say this because when I disable installing docker on kubernetes-master in vagrant, I do not have an issue. - is there a different way we can structure this on the kubernetes-master? For example, could we run the script in or something?


Question: func (rh *realHistory) ReleaseControllerRevision(parent metav1.Object, revision *apps.ControllerRevision) (*apps.ControllerRevision, error) { // Use strategic merge patch to add an owner reference indicating a controller ref <del> released, err := rh.client.Apps().ControllerRevisions(revision.GetNamespace()).Patch(revision.GetName(), </del> <ins> released, err := rh.client.AppsV1beta1().ControllerRevisions(revision.GetNamespace()).Patch(revision.GetName(), </ins> types.StrategicMergePatchType, []byte(fmt.Sprintf(`{"metadata":{"ownerReferences":[{"$patch":"delete","uid":"%s"}],"uid":"%s"}}`, parent.GetUID(), revision.UID)))

The statefulset controller and controller history implementations should be migrated off of the Apps client (deprecated) and onto the AppsV1BetaClient. Currently there is no user facing issue caused by the use of the deprecated resource, but we should maintain good code hygiene.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/assign kow3ns
/assign kow3ns
/cc
/assign


Question: var sshOptionsMap map[string]string <del> const archiveName = "e2e_node_test.tar.gz" </del> <ins> const ( archiveName = "e2e_node_test.tar.gz" CNI_RELEASE = "c864f0e1ea73719b8f4582402b0847064f9883b0" ) var CNI_URL = fmt.Sprintf("https://storage.googleapis.com/kubernetes-release/network-plugins/cni-%s.tar.gz", CNI_RELEASE) </ins> var hostnameIpOverrides = struct { sync.RWMutex

Doesn't seem to haven been caused by my change:
/cc
The test only failed on coreos, and the is full of error messages. Plus the log is almost unreadable because of the extremely high verbosity. I filed for that. shows that the node has been rebooted a couple of times. And the kernel log shows This looks like . The hairpin mode is set to hairpin-veth. /cc
Yeah don't run with hairpin-mode I guess. is coreos using kubenet?
Node problem detector should detect this issue and report KernelDeadLock condition.
Yeah, but this is node e2e test, we don't have node problem detector running on node e2e cluster.
Not sure how to assign coreos issues... Assigning to people to reassign based on thread activity. Please reassign correctly!
ping!
/cc
This happens in node e2e test right? I believe node e2e test is NOT using kubenet.
Running node e2e with kubenet+hairpin-veth could prevent this.
Sorry, this is lost in github notifications.. Feel free to ping me on slack if I am not responding in a day next time :)
To run kubenet on a node, the framework needs to download and unpack the cni plugin in kubelet needs to run with and a fixed pod CIDR (since there is no node controller) I did those in , but it failed on the trusty images because is not installed. We need to either install this in the node e2e framework, or bake it in to the image. what's the best practice for installing packages on the images?
I believe updating is the best way to bake it into the image, and I also think that should be preferred over installing it each time. You could do the same with CNI. Thanks for working on this!
I updated the PR to install nsenter on the trusty images. As for cni, I still keep it in the node e2e framework to install it every time because 1) it only downloads from GCS and untar a file of ~14MB, 2) the tarball is maintained and updated by It'd be an extra burden to update the images just for cni updates.
OK with the plan.

Seeing the error message again after switching to kubenet+hairpin-veth do you want to take a look at the logs?
let me take a look
urrrh... we hit it again. Switching back to
There's a difference in the above kernel log; the device is not a interface but rather an interface (presumably the one within a container's net ns since there is no eth0 on the host). I think all the prior instances of this were veths on the host.
True. This one is slightly different. I think deadlock on interface also showed up before.
Haven't seen any new occurrence since landed. We'll know in a day or so whether this problem has been fixed.
AFAIK, there are no new failures after landed. It should be safe to let this issue stay closed.
unfortunately I hit the bug with hairpinmode sets to promiscuous-bridge kubelet v1.3.0, linux kernel 4.6.3 cbr0 has already enter the promiscuous mode, all veth hairpinmode = 1 start with parameter
can you ? what distro are you running?
also are you using a network-plugin, overlay (like flannel), which cloudprovider?
Are the SELinux warnings normal ?
Maybe a little out of topic. Are you running K8s on GCE? Do you have the node problem detector running in your cluster?
all veths hairpin_mode = 1 my distro is coreos 1068.6.0 I don't use any network-plugins, and no cloudprovider support here, controller-manager start with kubelet setup after address allocated by controller-manager, all pods connected each other with static routes which controlled by ospfd running inside docker. I can confirm the selinux warnings not related to the bug.
not running on GCE. no detector running inside.
Alright, just want to verify whether the NPD works as expected.
here is the logs of kubelet startup.
That's the problem, the reason we ran into this to begin with is because of hairpin mode. Apparently it is still getting set, or isn't getting erased in your environment. I don't think we erase existing values.
that's wired.

I checked all my nodes, each veth hairpin_mode is 1. it seems it was set to 1 regardless of --configure-cbr0.
there is no existing veth before kubelet create bridge , docker could not start without cbr0 which created by kubelet.
hairpin_mode is always enabled if the device is a bridge interface,
Trust me it's not on my (and our other e2e test) clusters, or we'd know because tests would fail, and teh default setup is docker running on cbr0 and using veths. In 1.3 we started using kubenet network plugin by default, but hairpin flag wasn't set since 1.2. I can trace the code path down when I have some time, but I suspect it's something to do with the way coreos is setup.
I trust you, I promise. I did some tests. None of results told me the flag is set by kubelet. I could not figure out what was happend to these veths. finally, I duplicate to , the only difference is hairepinEnable default to "0", and invoke the hairpinunset.SetUpContainerPid when false. It works! But, if I create veth by hand, the default hairpin_mode is 0 , oh my god. It will not happen in my test environment, only in production env
will enable hairpin_mode That's my fault, I should not set it to false because kubernetes do not use docker's network model.


Question: "//vendor/github.com/prometheus/client_golang/prometheus:go_default_library", "//vendor/gopkg.in/gcfg.v1:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", <ins> "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/types:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", <ins> "//vendor/k8s.io/client-go/kubernetes:go_default_library", "//vendor/k8s.io/client-go/kubernetes/scheme:go_default_library", "//vendor/k8s.io/client-go/kubernetes/typed/core/v1:go_default_library", "//vendor/k8s.io/client-go/tools/record:go_default_library", </ins> ], )

On AWS when we call "attach" of a volume then we periodically we check (via API call) if volume has indeed been attached. We run the check in a loop with timeout of 21 minutes. AWS documents that - if a volume is stuck in "attaching" state for too long, the node where volume is being attached is in "dirty" state and must be rebooted to fix the problem. Until node is rebooted, more pods with volumes to same node is only going be stuck in same state (i.e "attaching"). We propose that - if attach operation times out waiting for volume in "attaching" state, we should simply taint the node so as it can not affect other pods and a admin can manually fix the node. /sig aws /sig storage cc


Question: persistentVolumeClaimColumnDefinitions := []metav1alpha1.TableColumnDefinition{ {Name: "Name", Type: "string", Format: "name", Description: metav1.ObjectMeta{}.SwaggerDoc()["name"]}, {Name: "Status", Type: "string", Description: apiv1.PersistentVolumeClaimStatus{}.SwaggerDoc()["phase"]}, <ins> {Name: "Volume", Type: "string", Description: apiv1.PersistentVolumeSpec{}.SwaggerDoc()["volumeName"]}, </ins> {Name: "Capacity", Type: "string", Description: apiv1.PersistentVolumeClaimStatus{}.SwaggerDoc()["capacity"]}, {Name: "Access Modes", Type: "string", Description: apiv1.PersistentVolumeClaimStatus{}.SwaggerDoc()["accessModes"]}, {Name: "StorageClass", Type: "string", Description: "StorageClass of the pvc"},

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: bug /kind bug What happened: returns the following: VOLUME label is missing. What you expected to happen: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.0.3928+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-08-18T01:09:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} /sig cli
/assign
GitHub didn't allow me to assign the following users: zjj2wry. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thank for you report this


Question: // - container is not an infrastructure (pause) container // - container is not already mounting on /etc/hosts // Kubernetes will not mount /etc/hosts if: <del> // - when the Pod sandbox is being created, its IP is still unknown. Hence, PodIP will not have been set. // - Windows pod contains a hostProcess container </del> <ins> // - the Pod is on the pod network and PodIP has not yet been set (e.g., Pod sandbox is being created). // - the Pod is on Windows, and contains a hostProcess container. </ins> func shouldMountHostsFile(pod *v1.Pod, podIPs []string) bool { <del> shouldMount := len(podIPs) > 0 </del> <ins> shouldMount := len(podIPs) > 0 || pod.Spec.HostNetwork </ins> if runtime.GOOS == "windows" { return shouldMount && !kubecontainer.HasWindowsHostProcessContainer(pod) }

When the node network is not yet connected, start kubelet and create static pod like this The created container file does not contain the configuration of hostAliases. The created container file should contain the configuration of hostAliases. kubelet service. static pod with configuration of hostAliases. network, like . kubelet service. created container file does not contain the configuration of hostAliases. No response <details</details<details</details<details</details<detailskubeadm </details<detailscontainerd 1.6.4 </details<details</details
/sig node
23 is EOL please check if this is happening in a supported release, latest supported is 1.25
You might want to set up a dependency on network from kubelet (assuming you're using systemd), so that it doesn't start until network is ready. Also, could you reproduce this on a later version (eg. 1.27)? /triage needs-information
The kubelet systemd configuration file already declares a network-online target dependency, but it doesn't seem to work. Looks like additional dependencies are required such as NetworkManager-wait-online.service. I think the best way is that kubelet is compatible with this scenario. Also, I also tested on 1.27 and the problem is still reproducible.
Is there any progress in dealing with this issue?
I haven't been working on this, not sure if anyone is actively looking into it.
did you try adding this dependency? Did it work?
NetworkManager-wait-online.service only waits for 60s. If the network is still abnormal after 60s, the problem will still occur.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network The kubelet wait for the PodIPs to be present to mount the etc hosts files however, it does not check if the pod is host network. If the Pod is using hostNetwork, then it seems the PodIPs are ignored so it seems legit to mount the aliases independently of PodIPs if the pod is using host network /assign /sig network /triage accepted
/remove-sig node


Question: } // Call SetUpDevice if blockVolumeMapper implements CustomBlockVolumeMapper <del> if customBlockVolumeMapper, ok := blockVolumeMapper.(volume.CustomBlockVolumeMapper); ok { </del> <ins> if customBlockVolumeMapper, ok := blockVolumeMapper.(volume.CustomBlockVolumeMapper); ok && actualStateOfWorld.GetDeviceMountState(volumeToMount.VolumeName) != DeviceGloballyMounted { </ins> var mapErr error stagingPath, mapErr = customBlockVolumeMapper.SetUpDevice() if mapErr != nil {

For raw block CSI volumes on Kubernetes, kubelet ends up calling CSI for every single "map" (i.e. raw block "mount") operation for a volume already attached to the node. , should be called once per volume per node: More specifically, . Create a raw block PVC for a CSI driver Create a pod using that PVC Create another pod using the same PVC and schedule to the same node Kubelet logs will show multiple for the same PVC. Example log: Problem appears to be here: This should first check if "volume is globally mounted" before executing . Does anyone know why we don't? For example, if you look a the equivalent code for file-on-block, it does the check: CC Repro'd on Kubernetes v1.23.5 N/A N/A N/A N/A CSI general (not specific to any driver)
/sig storage
CC -- thanks for your help
I think the reasoning above from is right. I traced the origin code is from here At that time both file and block will call MountDevice repeatedly without checking GetDeviceMountState from actual state. Later there is a fix to check the state, but it only put for file volume, not the block. So I think we should do the same for block here. WDYT?
Yes, NodeStage should be ideally called only once.
Hey Can I work on it? I think this is a cc
Yes please go ahead. We can help review.
/assign
/triage accepted
Thanks


Question: nodesNeedingDaemonPods = append(nodesNeedingDaemonPods, nodesNeedingDaemonPodsOnNode...) podsToDelete = append(podsToDelete, podsToDeleteOnNode...) <del> failedPodsObserved += failedPodsObservedOnNode </del> } // Remove unscheduled pods assigned to not existing nodes when daemonset pods are scheduled by scheduler.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I have a daemonsets with 250 replicas and maxUnavailable is 20%. When I try to update its image, the ds will scale down 50 old replicas and wait for SatisfiedExpectations, after that it will scale up 50 new replicas. But if a pod delete failed (node is notready, and pod stuck at terminating status), ds controller will never reach the expectation and just stuck here. What you expected to happen: When a pod stuck at terminating, ds controller should start new pods like deployment controller, instead of waiting for the delete batch finish and ready for next poll. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.1-8+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-25T13:04:04Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig apps /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
DaemonSet controller waits for old pods to terminate before creating new ones for best-effort at-most-one pod per node. This is a deliberate design decision.

In most cases, DaemonSets won't tolerate two pods scheduled on the same node at the same time; for example, it's common to communicate with DaemonSet pods via . I'm going to close this ticket as this works as designed. Please feel free to reopen the ticket if you think there are still issues to be resolved.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
why does the DaemonSet controller wait for all pods 50 to be done, before moving to the NEXT 50 ? I completely understand that we want only at most 1 pod per node. But if a pod deletion fails on the node, move on to the next pod as long as the total unavailable pods are less than maxUnavailable. This will allow us to make progress in the face of bad nodes or failed pod deletions due to failing tear down of bad volumes or other reasons.
/assign
thoughts on this one ?
Any progress on this issue? We also encountered this problem. There are thousands of nodes in our prod cluster and nodes often go down, which cause daemonset rolling update stucked, this problem really bother us.
PTAL
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ginkgo.It("should function for pod-Service: sctp [Feature:SCTPConnectivity][Disruptive]", func() { config := e2enetwork.NewNetworkingTestConfig(f, false, true) ginkgo.By(fmt.Sprintf("dialing(sctp) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterSCTPPort)) <del> config.DialFromTestContainer("sctp", config.ClusterIP, e2enetwork.ClusterSCTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err := config.DialFromTestContainer("sctp", config.ClusterIP, e2enetwork.ClusterSCTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> ginkgo.By(fmt.Sprintf("dialing(sctp) %v --> %v:%v (nodeIP)", config.TestContainerPod.Name, config.NodeIP, config.NodeSCTPPort)) <del> config.DialFromTestContainer("sctp", config.NodeIP, config.NodeSCTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err = config.DialFromTestContainer("sctp", config.NodeIP, config.NodeSCTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> }) ginkgo.It("should function for node-Service: http", func() { config := e2enetwork.NewNetworkingTestConfig(f, true, false) ginkgo.By(fmt.Sprintf("dialing(http) %v (node) --> %v:%v (config.clusterIP)", config.NodeIP, config.ClusterIP, e2enetwork.ClusterHTTPPort)) config.DialFromNode("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) ginkgo.By(fmt.Sprintf("dialing(http) %v (node) --> %v:%v (nodeIP)", config.NodeIP, config.NodeIP, config.NodeHTTPPort)) config.DialFromNode("http", config.NodeIP, config.NodeHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) })

What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: cmd.wait: - name: /opt/kubernetes/helpers/services bounce kubelet - watch: <ins> - file: /var/lib/kubelet/pki </ins> - file: /usr/local/bin/kubelet - file: {{ pillar.get('systemd_system_path') }}/kubelet.service - file: {{ environment_file }}

https://k8s- Judging by the timestamp, it's this batch of merge caused the failure: NONE
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
looks like it's your pr causing the failure ^^^
cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
It's still failing, See build log: https://k8s-, kubelet log: error in kubelet log:


Question: ctrl := gomock.NewController(t) defer ctrl.Finish() <del> az := GetTestCloud(ctrl) mockLBClient := az.LoadBalancerClient.(*mockloadbalancerclient.MockInterface) mockLBClient.EXPECT().List(gomock.Any(), az.ResourceGroup).Return(nil, &retry.Error{HTTPStatusCode: http.StatusInternalServerError}) </del> <ins> tests := []struct { clientErr   *retry.Error expectedErr error }{ { clientErr:   &retry.Error{HTTPStatusCode: http.StatusInternalServerError}, expectedErr: fmt.Errorf("Retriable: false, RetryAfter: 0s, HTTPStatusCode: 500, RawError: <nil>"), }, { clientErr:   &retry.Error{HTTPStatusCode: http.StatusNotFound}, expectedErr: nil, }, } for _, test := range tests { az := GetTestCloud(ctrl) mockLBClient := az.LoadBalancerClient.(*mockloadbalancerclient.MockInterface) mockLBClient.EXPECT().List(gomock.Any(), az.ResourceGroup).Return(nil, test.clientErr) pips, err := az.ListLB(&v1.Service{}) assert.Equal(t, test.expectedErr, err) assert.Empty(t, pips) } </ins> <del> pips, err := az.ListLB(&v1.Service{}) assert.Equal(t, fmt.Errorf("Retriable: false, RetryAfter: 0s, HTTPStatusCode: 500, RawError: <nil>"), err) assert.Empty(t, pips) </del> } func TestListPIP(t *testing.T) { ctrl := gomock.NewController(t) defer ctrl.Finish() <del> az := GetTestCloud(ctrl) mockPIPClient := az.PublicIPAddressesClient.(*mockpublicipclient.MockInterface) mockPIPClient.EXPECT().List(gomock.Any(), az.ResourceGroup).Return(nil, &retry.Error{HTTPStatusCode: http.StatusInternalServerError}) </del> <ins> tests := []struct { clientErr   *retry.Error expectedErr error }{ { clientErr:   &retry.Error{HTTPStatusCode: http.StatusInternalServerError}, expectedErr: fmt.Errorf("Retriable: false, RetryAfter: 0s, HTTPStatusCode: 500, RawError: <nil>"), }, { clientErr:   &retry.Error{HTTPStatusCode: http.StatusNotFound}, expectedErr: nil, }, } for _, test := range tests { az := GetTestCloud(ctrl) mockPIPClient := az.PublicIPAddressesClient.(*mockpublicipclient.MockInterface) mockPIPClient.EXPECT().List(gomock.Any(), az.ResourceGroup).Return(nil, test.clientErr) </ins> <del> pips, err := az.ListPIP(&v1.Service{}, az.ResourceGroup) assert.Equal(t, fmt.Errorf("Retriable: false, RetryAfter: 0s, HTTPStatusCode: 500, RawError: <nil>"), err) assert.Empty(t, pips) </del> <ins> pips, err := az.ListPIP(&v1.Service{}, az.ResourceGroup) assert.Equal(t, test.expectedErr, err) assert.Empty(t, pips) } </ins> } func TestCreateOrUpdatePIP(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I recall that this issue was reported previously in It was fixed in the master back then with and it was cherry-picked into with However we still observe the same issue with the legacy azure cloud-controller-manager v1.19.9. legacy azure cloud controller-manager to be able to delete LoadBalancer when the underlying resource group is deleted. a LoadBalancer manually the underlying resource group the LoadBalancer created in step 1 sure that the deletion of the LoadBalancer fails Events of the Service: Kubernetes version (use ): v1.19.9 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure /cc
Thanks for reporting the issue. It should be a bug inside cloud provider. /triage accepted


Question: return true } <ins> func deepCopyPrometheusLabels(m prometheus.Labels) prometheus.Labels { ret := make(prometheus.Labels, len(m)) for k, v := range m { ret[k] = v } return ret } </ins>

Currently the kubelet exports liveness/readiness probe metric information at an . The metric data looks like this: This is incredibly useful metric information, however, it would be better if these metrics were counters rather than gauges. Currently a success probe is registered as a '0' value and a failure is registered as a '1' value. However, depending on your prometheus scrape interval, you can lose bits of information. It would be better if we emitted these metrics as counters. /sig instrumentation node
Perhaps a new counters metrics to record the the number of failure is a better option. Because the gauge metrics is easy to watch and alert. I can do it if agree with it. /assign
This is a good point. However, if we really did want to use it for this purpose, it would be better if the gauge metric reported '0' for success or else the current running count of failures for a given prober endpoint. With the current implementation, we are needlessly dropping some bits. It may be useful to have both. One can provide a more real time signal for alerting but the other gives you better historical data on master component probe results.
do you have a preference on gauge vs counter here?
For our purposes a counter makes much more sense, because we want to use it to figure out the proportion of time something was healthy (e.g. for calculating availability SLOs). It also means we can have a much higher fidelity measurement - e.g. we could ramp up the healthchecks to every 5s but only scrape the results every minute. Can you point at some examples of people doing this, so I can better understand how these metrics are being used? It seems like using point-in-time samples of health is going to result in noisy alerts, unless you do more work elsewhere to aggregate ("alert if this metric was 0 for X minutes" - which then runs the risk of missing samples, as mentioned)
if the metrics value is 0 or 1, we can simple set prometheus rule as just like you mentioned.
Yes, I agree that the alert rule would be simple but it also feels prone to be noisy (as mentioned). A failed probe is not necessarily alert-worthy, since the kubelet defers action until the ; by default, the kubelet doesn't do anything until we have 3 consecutive failures.

Yep, so with in alert rule, it will not fire alert noisily. /cc
All the facts have already been stated, I'm in favor of removing the gauges and add counters (I don't see a state that we can't interpolate from the counter that the gauge displays today, yet the counter is more reliable in alerting aspects as mentioned).
sure. As a counter, how to represent successful probe? If the probe is successful, no action is taken, otherwise plus 1 to this metrics ?
I'm thinking something along the lines of:
Yeah, I was thinking along the same lines (i.e. using the result of the probe as a label).
Also now that we're talking about this and inspired by the metrics overhaul of 1.14, we should probably standardize these label values across Kubernetes. I'll add it to the next agenda.
A little off topic but wondering if someone could help guide me on how to access this endpoint? I've tried both k8s 1.13 and 1.14 and cannot access metrics/probes (i can only hit /metrics) via the api server http endpoint. Is it exposed somewhere else, or is it a flag I have to enable?
I had this issue for few days. What you should do is in your config you have to add "metricspath: /metrics/cadvisor". This will scrape out all container metrics that you would like to see. global: scrapeinterval: 5s evaluationinterval: 5s rulefiles: /etc/prometheus-rules/cpu- /etc/prometheus-rules/instance- scrapeconfigs: jobname: 'kubernetes-nodes' tlsconfig: cafile: bearertokenfile: kubernetessdconfigs: role: node metricspath: /metrics/cadvisor relabelconfigs: sourcelabels: [address] regex: '(.*):' replacement: '${1}:' targetlabel: address
This isn't an apiserver metric, it's actually a kubelet metric, so it's exposed on a different port. Are you using 8080? I think it should be or something like that.
@ munsayac13 thanks for the suggestion.

This is what I ended up doing to solve this: update the kubelet servicemonitor (e.g kubectl edit servicemonitor kubelet) this: bearerTokenFile: honorLabels: true interval: 30s path: /metrics/probes port: https-metrics scheme: https tlsConfig: insecureSkipVerify: true
Hi folks, I wonder if we could add the old gauge metrics back for backward compatibility sake. We just realized this change breaks some of our components that relies on the old metrics during the migration to v1.15.x.
We don’t have stability guarantees on these metrics (or any for that matter). I don’t think we’ll be reintroducing these, but we’re introducing a metrics stability framework for these things to not happen again or have the same deprecation process as kubernetes APIs in the future. Sorry for the inconvenience!
that is understandable.