Question: if err != nil { framework.Failf("error waiting for pod %s to be unready %v", webserverPod0.Name, err) } <del> // Wait the change has been propagated and the service start to fail clusterIPAddress := net.JoinHostPort(svc.Spec.ClusterIP, strconv.Itoa(servicePort)) cmd := fmt.Sprintf(`curl -q -s --connect-timeout 5 %s/hostname`, clusterIPAddress) if pollErr := wait.PollImmediate(framework.Poll, e2eservice.KubeProxyEndpointLagTimeout, func() (bool, error) { </del> <ins> nodeIPs0 := e2enode.GetAddresses(&node0, v1.NodeInternalIP) nodeIPs1 := e2enode.GetAddresses(&node1, v1.NodeInternalIP) nodePortAddress0 := net.JoinHostPort(nodeIPs0[0], strconv.Itoa(int(svc.Spec.Ports[0].NodePort))) nodePortAddress1 := net.JoinHostPort(nodeIPs1[0], strconv.Itoa(int(svc.Spec.Ports[0].NodePort))) // Wait until the change has been propagated to both nodes. cmd := fmt.Sprintf(`curl -q -s --connect-timeout 5 %s/hostname`, nodePortAddress0) if pollErr := wait.PollUntilContextTimeout(ctx, framework.Poll, e2eservice.KubeProxyEndpointLagTimeout, true, func(_ context.Context) (bool, error) { _, err := e2eoutput.RunHostCmd(pausePod1.Namespace, pausePod1.Name, cmd) if err != nil { return true, nil } return false, nil }); pollErr != nil { framework.ExpectNoError(pollErr, "pod on node0 still serves traffic") } cmd = fmt.Sprintf(`curl -q -s --connect-timeout 5 %s/hostname`, nodePortAddress1) if pollErr := wait.PollUntilContextTimeout(ctx, framework.Poll, e2eservice.KubeProxyEndpointLagTimeout, true, func(_ context.Context) (bool, error) { </ins> _, err := e2eoutput.RunHostCmd(pausePod1.Namespace, pausePod1.Name, cmd) if err != nil { return true, nil } return false, nil }); pollErr != nil { <del> framework.ExpectNoError(pollErr, "service still serves traffic") </del> <ins> framework.ExpectNoError(pollErr, "pod on node1 still serves traffic") </ins> } <del> nodeIPs0 := e2enode.GetAddresses(&node0, v1.NodeInternalIP) nodeIPs1 := e2enode.GetAddresses(&node1, v1.NodeInternalIP) nodePortAddress0 := net.JoinHostPort(nodeIPs0[0], strconv.Itoa(int(svc.Spec.Ports[0].NodePort))) nodePortAddress1 := net.JoinHostPort(nodeIPs1[0], strconv.Itoa(int(svc.Spec.Ports[0].NodePort))) </del> <ins> clusterIPAddress := net.JoinHostPort(svc.Spec.ClusterIP, strconv.Itoa(servicePort)) </ins> // connect 3 times every 5 seconds to the Service and expect a failure for i := 0; i < 5; i++ { cmd = fmt.Sprintf(`curl -q -s --connect-timeout 5 %s/hostname`, clusterIPAddress)

also see it flaky in /kind flake /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hmm, it does not look well indeed , I don't have time immediately but appreciate if someone can do an analysis of one of the failures /cc
failing jobs are not storing the kube-proxy logs, this is critical to solve ASAP or we are blind on failures
is merged to fix that. Let's wait for the next flake to happen.
Got 1 in the kube-proxy log is available
ok, this goes this way The only verifies that the change has been propagated to the specific node where that is tested , however, later we test nodePorts on different nodes, and kube-proxies in different nodes have different propagation times See The test pod runs in bootstrap-e2e-minion-group-sdr1, that is 10.138.0.6 138.0.3 is bootstrap-e2e-minion-group-v9j8, if we check , we can observe that the service is not processed until 3 seconds later than the check 00:39:09 vs 00:39:06 The solution is to validate on both nodes we are doing the check that kube-proxy has programmed the rules, or just validate one node Important, kube-proxy on gce jobs has a 10s period sync delay, so changes are accumulated during that period of 10s and flushed at same time to improve scale
/help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met.

If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Hi! I would like to help with this. For now, I am trying to understand how the test works, but I don’t get something. So two pod configurations are created, one with a webserver and one with an agn image. The first one is assigned to node0 and the second one to node1. From the agn pod, which is on node1, curls are done to node0 to the webserver: First with clusterIPaddres:Second with the nodePortAddress0 But why is the third one needed? Isn’t this a curl from the pod from node0 to itself? Is this relevant to the test? Or am I missing something?
to exercise that it works from same node to cluster ip, same node to nodeport, node to nodeport on other node Those are different paths
I made a PR. When you have a moment to check it, let me know if it's okay or if anything needs to be modified. Thank you!


Question: for i := range replicas { // delete and recreate failed pods if isFailed(replicas[i]) { <del> glog.V(4).Infof("StatefulSet %s/%s is recreating failed Pod %s", </del> <ins> ssc.recorder.Eventf(set, v1.EventTypeWarning, "RecreatingFailedPod", "StatefulSet %s/%s is recreating failed Pod %s", </ins> set.Namespace, set.Name, replicas[i].Name)

An issue to make bot happy for When StatefulSet finds any of its pods in phase Failed it will delete it and create it again. While doing so it will emit delete+create event which is really confusing for the user as he has no idea what's happening. /sig apps


Question: mountHeader: mountSecretNames, tokenHeader: tokenSecretNames, } <del> for header, names := range types { </del> <ins> for _, header := range sets.StringKeySet(types).List() { names := types[header] </ins> if len(names) == 0 { w.Write(LEVEL_0, "%st<none>n", header) } else {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: unit test fail randomly. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I am working on it.
/sig cli
/kind bug
Thanks for opening the issue. I'm experiencing this too.


Question: func (ep *endpointsInfo) Cleanup() { Log(ep, "Endpoint Cleanup", 3) <del> if ep.refCount != nil { </del> <ins> if !ep.GetIsLocal() && ep.refCount != nil { </ins> *ep.refCount-- // Remove the remote hns endpoint, if no service is referring it

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --kube-proxy panics when a LoadBalancer service is created after a deployment it is associated with is created and the pods are fully running. kube-proxy does not panic. 1) Create 2 Windows nodes 2) Create a deployment, ensuring that the pods are scheduled on different nodes: 3) Wait for the pods to go into the state. kube-proxy will not crash if the pods are not fully deployed 4) Create a LoadBalancer service for the deployment 5) At this point kube-proxy should have crashed. If not, delete and recreate the LoadBalancer, this should trigger the panic. The kube-proxy exe was run as: Stdout/err shows: Full kube-proxy logs: Kubernetes version (use ): 1.20 Cloud provider or hardware configuration: AWS OS (e.g: ): Windows server 2019 Kernel (e.g. ): 10.0..1817 Install tools: OpenShift 4.7.3 Network plugin and version (if this is a network-related bug): Hybrid OVN Others:
/sig windows
are you aware of anything happening here?
I am taking a look at this.
/triage accepted


Question: return nil } <ins> var strictDecodingErr error </ins> obj, parsedGvk, err := d.Decode(c.Args.Raw, &gvk, nil) if err != nil { <del> return fmt.Errorf("decoding args for plugin %s: %w", c.Name, err) </del> <ins> decodingArgsErr := fmt.Errorf("decoding args for plugin %s: %w", c.Name, err) if obj != nil && runtime.IsStrictDecodingError(err) { strictDecodingErr = runtime.NewStrictDecodingError([]error{decodingArgsErr}) } else { return decodingArgsErr } </ins> } if parsedGvk.GroupKind() != gvk.GroupKind() { return fmt.Errorf("args for plugin %s were not of type %s, got %s", c.Name, gvk.GroupKind(), parsedGvk.GroupKind()) } c.Args.Object = obj <del> return nil </del> <ins> return strictDecodingErr </ins> } func (c *PluginConfig) encodeNestedObjects(e runtime.Encoder) error {

With the addition of server side validation in 1.23, it is now possible for a call to successfully decode it’s data and return a non-nil error when there are strict decoding errors (e.g. duplicate/unknown fields). This means that callers of can distinguish between fatal errors (that cause decoding to fail) and strict decoding errors (that still allow the decode to succeed) by inspecting the target object (to see if it was successfully decoded into) and the error itself (to see if it is a ) With the NestedObjectDecoder however, it only returns an error from giving no indication whether a strict decoding error it has returned comes from an internal decode step that short circuited (by not accounting for strict decoding errors) or actually did successfully decode, but also encountered strict decoding errors. We propose taking a few actions to minimize the possibility of erroneous short-circuiting from Document how all nested decoders should only return strict decoding errors if all decoding is successful or return only non-strict decoding (ie. fatal errors). Document this for all known in-tree and out-of-tree . Update in-tree nested decoders to do the above mentioned behavior (which is just the kube-scheduler) Long-term solutions to better guarantee that no erroneous short-circuiting is taking place could include deprecating the current interface in favor of one that returns both the target object that was decoded into along with an error return a special error type specific to indicating that all internal decoding was successful.
/sig api-machinery
/triage accepted


Question: conformImages = append(conformImages, conformImage) } <del> By("not presenting images [Conformance]", func() { </del> <ins> By("not presenting images", func() { </ins> for _, conformImage := range conformImages { present, err := conformImage.Present() Expect(err).ShouldNot(HaveOccurred())

As of now node e2e requires all the tests to pass. There are tests like the container conformance tests which have been flaky in the past - , , We need ability to mark certain tests as flaky and not run them as part of merge queue.
Ginkgo flags makes it possible to skip tests. We can use a prefix to not run flaky tests in our existing jenkins CIs.
Closing via


Question: return NewKubectlCommand(namespace, args...).Exec() } <ins> // RunKubectlWithFullOutput is a convenience wrapper over kubectlBuilder // It will also return the command's stderr. func RunKubectlWithFullOutput(namespace string, args ...string) (string, string, error) { return NewKubectlCommand(namespace, args...).ExecWithFullOutput() } </ins> // RunKubectlOrDieInput is a convenience wrapper over kubectlBuilder that takes input to stdin func RunKubectlOrDieInput(namespace string, data string, args ...string) string { return NewKubectlCommand(namespace, args...).WithStdinData(data).ExecOrDie(namespace)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-windows jobs: Which test(s) are failing: [sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance] [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] Since when has it been failing: It was a false positive until recently. Testgrid link: Reason for failure: Partially Qualified Domain Names are not currently supported on Windows containers. The tests are checking PQDNs for validation. The test was passing until recently because was still returning the exit code 0, even if the given DNS name was not found: [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] [BeforeEach] [sig-network] Services STEP: Creating a kubernetes client Apr 28 21:24:38.587: INFO: kubeConfig: /home/ubuntu/kube/config200424 STEP: Building a namespace api object, basename services STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in services-2633 STEP: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-network] Services [It] should be able to change the type from NodePort to ExternalName [Conformance] STEP: creating a service nodeport-service with the type=NodePort in namespace services-2633 STEP: Creating active service to test reachability when its FQDN is referred as externalName for another service STEP: creating service externalsvc in namespace services-2633 STEP: creating replication controller externalsvc in namespace services-2633 I0428 21:24:42. ] Created replication controller with name: externalsvc, namespace: services-2633, replica count: 2 I0428 21:24:45. ] externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:48. ]

externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:51. ] externalsvc Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:54. ]

externalsvc Pods: 2 out of 2 created, 2 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady STEP: changing the NodePort service to type=ExternalName Apr 28 21:24:55.544: INFO: Creating new exec pod Apr 28 21:25:06.371: INFO: Running '/home/ubuntu/rune2e/kubernetes/output/dockerized/bin/linux/amd64/kubectl --server=https://kubetest--83ef-11ea-ae60- --kubeconfig=/home/ubuntu/kube/config200424 exec --namespace=services-2633 agnhost-pod -- /bin/sh -x -c nslookup nodeport-service' Apr 28 21:25:09.610: INFO: stderr: "+ nslookup nodeport-servicen kube- can't find nodeport-service: Non-existent domainrrn" Apr 28 21:25:09.610: INFO: stdout: "Server: kube-rnAddress: 10.0.0.10rnrn" STEP: deleting ReplicationController externalsvc in namespace services-2633, will wait for the garbage collector to delete the pods Apr 28 21:25:10.429: INFO: Deleting ReplicationController externalsvc took: 423.628373ms Apr 28 21:25:10.830: INFO: Terminating ReplicationController externalsvc pods took: 400.686463ms Apr 28 21:25:15.792: INFO: Cleaning up the NodePort to ExternalName test service [AfterEach] [sig-network] Services Apr 28 21:25:16.230: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready STEP: Destroying namespace "services-2633" for this suite. [AfterEach] [sig-network] Services â¢{"msg":"PASSED [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance]","total":2,"completed":2,"skipped":2913,"failed":0} The issue was observed with an updated, which doesn't return 0 if the given DNS name was not found. Anything else we need to know*: /sig testing /sig windows


Question: return "", "", err } <ins> // ResizeDisk Expand the disk to new size func (c *ManagedDiskController) ResizeDisk(diskName string, oldSize resource.Quantity, newSize resource.Quantity) (resource.Quantity, error) { ctx, cancel := getContextWithCancel() defer cancel() result, err := c.common.cloud.DisksClient.Get(ctx, c.common.resourceGroup, diskName) if err != nil { return oldSize, err } if result.DiskProperties == nil || result.DiskProperties.DiskSizeGB == nil { return oldSize, fmt.Errorf("DiskProperties of disk(%s) is nil", diskName) } requestBytes := newSize.Value() // Azure resizes in chunks of GiB (not GB) requestGiB := int32(util.RoundUpSize(requestBytes, 1024*1024*1024)) newSizeQuant := resource.MustParse(fmt.Sprintf("%dGi", requestGiB)) glog.V(2).Infof("azureDisk - begin to resize disk(%s) with new size(%d), old size(%v)", diskName, requestGiB, oldSize) // If disk already of greater or equal size than requested we return if *result.DiskProperties.DiskSizeGB >= requestGiB { return newSizeQuant, nil } result.DiskProperties.DiskSizeGB = &requestGiB ctx, cancel = getContextWithCancel() defer cancel() if _, err := c.common.cloud.DisksClient.CreateOrUpdate(ctx, c.common.resourceGroup, diskName, result); err != nil { return oldSize, err } glog.V(2).Infof("azureDisk - resize disk(%s) with new size(%d) completed", diskName, requestGiB) return newSizeQuant, nil } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: According to , some volumes(e.g. aws, nfs, ceph) has already supported PV resizing, need to add PV resize support for azure disk What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: PV resize doc is here: Update: For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I will try to work on resizing an azure disk which is not in attached state. May dig into more details later. Environment: Kubernetes version (use ): v1.10.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
Hi , is PV size grow feature still in alpha in master now? How do I open this feature, set in kubelet, kube-apiserer and kube-controller-manager config?
I have figured out that should be set in kubelet, kube-apiserer, kube-controller-manager, kube-scheduler config, and also need to enable in apiserver admission control. It works well.
should we go ahead and close this issue?
also file needs to be updated to allow resizing. This plugin provides additional checks for resizing.
For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I would suggest work on first, could you review , it works well in my testing env.
Adding priority important-soon to this issue so that it can stay in 1.10. If that is incorrect, please update. priority important-soon
/priority important-soon
PR has been merged, can this issue be closed?
yup, looks like it's in. Thanks for flagging! /close
you can't close an active issue unless you authored it or you are assigned to it, Can only assign issues to org members and/or repo collaborators.. <detailsIn response to : yup, looks like it's in. Thanks for flagging! /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Awww shucks, looks like I don't have magic powers. please close this when you have a chance?
has been merged, while for azure disk work is just in investigation state mainly because there is blocking issue: it does not support disk resize on a Running VM.
could someone move this issue out of milestone v1.10, thanks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Why was this closed? This is not fixed by that crappy PR. This is the kubernetes repo, and this refers to k8s apis being implemented correctly iyam. The PR that you closed this with does NOT support disk resize on a Running VM. Your words Please reopen this.
pls track this issue for online resize feature:


Question: const ( // resyncPeriod for informer // TODO (https://github.com/kubernetes/kubernetes/issues/123688): disable? <del> resyncPeriod = time.Duration(10 * time.Minute) </del> <ins> resyncPeriod   = time.Duration(10 * time.Minute) retryPeriod    = 5 * time.Second maxRetryPeriod = 180 * time.Second backoffFactor  = 2.0 // Introduce a backoff multiplier as jitter factor </ins> ) // nodeResourcesController collects resource information from all registered

has to retry after failures. This currently uses a fixed 5 second delay to avoid busy-looping. Exponential backoff might be better. /sig node /triage-accepted /priority important-longterm /lifecycle frozen /assign
/triage accepted
/assign


Question: func TestPriorityAndFairnessWithPanicRecoveryAndTimeoutFilter(t *testing.T) { epmetrics.Register() fcmetrics.Register() <ins> timeFmt := "15:04:05.999" </ins> t.Run("priority level concurrency is set to 1, request handler panics, next request should not be rejected", func(t *testing.T) { const (

pull-kubernetes-unit TestPriorityAndFairnessWithPanicRecoveryAndTimeoutFilter 7/14 No response /sig api-machinery
/priority important-soon
/cc /triage accepted
and can this be related to ?
I see no direct relationship, but hey, anything's possible. We learned that with the misery in integration test failures when we merged not-really-guilty other stuff.
The expanded logging turned up some new evidence. See , where the complaint is now
The problem here is not APF, it is something in the timeout filter or the test server.
is something in APF triggering a timeout?
The flake definitely came in with on the prior commit on master: on the merge commit of
/assign
Note that the flaking test is deliberately suffering server-side timeouts. The symptom in flakes is that the test is checking that the client gets kube's designed response to a timeout while handling, a particular form HTTP response message, but sometimes the client gets something else (an http2 internal error?). is a change in metrics, and should have nothing to do with how timeouts while handling and detected or handled. I will see what I can see...
, regarding your comment above (): note that the output shown (the panic "as designed") is normal; if there are failures involved, the failing part was elided.
Also, the failures seem to be coming from one particular subtest () and that one does not involve designed panics.
On closer examination of the timeout filter, it appears that this "INTERNAL ERROR" is something that can happen normally in the course of handling a timeout. So I will adjust the test to accept this.
I have updated to include the fix for this flake.


Question: # Restart the kube-proxy on master ($1) function restart-apiserver() { echo "... in restart-kube-apiserver()"  >&2 <del> ssh-to-node "$1" "sudo /etc/init.d/kube-apiserver restart" </del> <ins> ssh-to-node "$1" "sudo docker kill `sudo docker ps | grep /kube-apiserver | awk '{print $1}'`" </ins> } # Execute after running tests to perform any required clean-up.  This is called

Now that kube-apiserver runs in a container, no longer exists: /cc
Actually fixed by .


Question: } // cleanupAssumedPods exists for making test deterministic by taking time as input argument. <ins> // It also reports metrics on the cache size for nodes, pods, and assumed pods. </ins> func (cache *schedulerCache) cleanupAssumedPods(now time.Time) { cache.mu.Lock() defer cache.mu.Unlock() <ins> defer cache.updateMetrics() </ins> // The size of assumedPods should be small for key := range cache.assumedPods {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a metric to track scheduler cache size. metric name Description -- -- schedulercachesize Number of elements in scheduler cache. Why is this needed: Gives better observability of scheduler cache. For, the number of "assumedpods" gives us an idea of how many pods are "assumed" and therefore being "bound". We can compare this with number of to make sure assume-bind works as expected. /sig scheduling
Is anyone already working on these metrics? If not I'd like to help out /assign
Thanks . Feel free to work on this one. Just a reminder, we may need feedback/approval from sig-instrumentation.


Question: package yaml import ( <ins> "fmt" "math" </ins> "reflect" <ins> "strconv" </ins> "testing" ) type MarshalTest struct { A string <ins> B int64 // Would like to test float64, but it's not supported in go-yaml. // (See https://github.com/go-yaml/yaml/issues/83.) C float32 </ins> } func TestMarshal(t *testing.T) { <del> s := MarshalTest{"a"} e := []byte("A: an") </del> <ins> f32String := strconv.FormatFloat(math.MaxFloat32, 'g', -1, 32) s := MarshalTest{"a", math.MaxInt64, math.MaxFloat32} e := []byte(fmt.Sprintf("A: anB: %dnC: %sn", math.MaxInt64, f32String)) </ins> y, err := Marshal(s) if err != nil {

I spin up a new cluster using GKE at version 0.9.2. Then I run: $kubectl get minion <minion-id-o yaml This returns a yaml file: apiVersion: v1beta1 creationTimestamp: 2015-02-03T01:16:46Z hostIP: <ipid: <minion-idkind: Node labels: diskSize: "massive" resourceVersion: 12 resources: capacity: cpu: "1" memory: 4.+09 selfLink: /api/v1beta1/minions/<minion-idstatus: conditions: kind: Ready lastTransitionTime: null status: Full uid: -ab42-11e4-9817- I delete the hostIp and status so that the update request won't get rejected ( should fix that), then run: $ kubectl update -f ERROR: F0202 18:08:19. 767 ] error unmarshaling JSON: json: cannot unmarshal number 4.+09 into Go value of type int I'm not sure whether the marshaling or the unmarshaling is what needs to be fixed, but the API shouldn't be returning numbers that it can't parse.
It looks like Quantity is unmarshalling wrong? , Alex Robinson wrote:
That's very likely. And sorry, I didn't realize yaml didn't paste into comments well. Here's the actual yaml:
Yeah, on the client-side, apparently, since the error is in It's possible that was missed during the Quantity refactoring. It may be troublesome that the v1beta1 API accepts a util.IntOrString for resources rather than a Quantity:
/cc
The (external) yaml library that we use is the cause for this. I will send a patch to fix the yaml library.
Sent a to our yaml dependency.
If we are only fixing our yaml unmarshalling, we are still losing precision. E.g. The yaml that Alex posted above is supposed to be representing 3.75 GiB. 75 GiB * 2^30 = , which we are representing as 4.+09. The amount of memory we are "losing" here is trivial, but if this is actually using scientific notation, we could be rounding up half the time as well (creating a very very low chance of an unexpected OOM).

Both of these are pretty small problems, but it seems silly to needlessly lose precision. Can we just not write these in scientific notation?
Resource quantities must be lossless. , CJ Cullen wrote:
+1. These are supposed to be strings, not integers nor floats.
What does return?
That does look like an API bug. It should be a string, IIUC.
Legacy CPU and memory are int and int64, respectively. Resource.Quantity should always produce strings.
This can't possibly work. IntOrString is a 32-bit signed int. This value doesn't fit in 32 bits.
Yes - this isn't a problem with the YAML library. This is a problem with trying to render a number larger than 32 bits () as a 32-bit integer (IntOrString), which instead ends up rendering as a floating point. (or something that keeps this value as a 64-bit integer the whole time) is the right fix.
The yaml library converts an into a as part of Marshal. Isn't that an issue with the yaml library? I agree that the kubernetes API has another issue which is that of not being able to represent resources greater than MaxInt. But to me these are two separate issues.
Even an int32 gets marshaled into a floating point number. I don't see how that is valid. The underlying golang package has separate APIs to address this behavior and the YAML library will also have to do that.
IntOrString stores things as "int", which is 32 or 64 bits depending on the platform. We should explicitly use int32 or int64 in our api objects. These values should be stored as a Quantity in v1beta3, and converted to a string in v1beta1 & 2.
Ack , Daniel Smith wrote:
If someone changed IntOrString to use int64 I would approve. , Vish Kannan wrote:
Assigned to myself.


Question: ct.ChangeColor(ct.Yellow, false, ct.None, false) fmt.Fprint(out, link) ct.ResetColor() <del> // TODO remove this warn once trailing slash is no longer required if warn { fmt.Fprint(out, " (note the trailing slash)") } </del> fmt.Fprintln(out, "") }

Right now at the end of I get We should update the output to point to the appropriate v1beta3 URL endpoints instead.
/cc
Ref and . cc
And as a minor nit, not all of those URLs need a trailing slash hack e.g. I don't think it is required for Elasticsearch. I think most recently worked on this code?
Thanks for fixing this


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package printers import ( "bytes" "strings" "testing" "k8s.io/apimachinery/pkg/runtime" api "k8s.io/kubernetes/pkg/apis/core" ) func TestTemplate(t *testing.T) { testCase := map[string]struct { template  string obj       runtime.Object expectOut string expectErr func(error) (string, bool) }{ "support base64 decoding of secret data": { template: "{{ .Data.username | base64decode }}", obj: &api.Secret{ Data: map[string][]byte{ "username": []byte("hunter"), }, }, expectOut: "hunter", }, "test error path for base64 decoding": { template: "{{ .Data.username | base64decode }}", obj:      &badlyMarshaledSecret{}, expectErr: func(err error) (string, bool) { matched := strings.Contains(err.Error(), "base64 decode") return "a base64 decode error", matched }, }, } for name, test := range testCase { buffer := &bytes.Buffer{} p, err := NewTemplatePrinter([]byte(test.template)) if err != nil { if test.expectErr == nil { t.Errorf("[%s]expected success but got:n %vn", name, err) continue } if expected, ok := test.expectErr(err); !ok { t.Errorf("[%s]expect:n %vn but got:n %vn", name, expected, err) } continue } err = p.PrintObj(test.obj, buffer) if err != nil { if test.expectErr == nil { t.Errorf("[%s]expected success but got:n %vn", name, err) continue } if expected, ok := test.expectErr(err); !ok { t.Errorf("[%s]expect:n %vn but got:n %vn", name, expected, err) } continue } if test.expectErr != nil { t.Errorf("[%s]expect:n errorn but got:n no errorn", name) continue } if test.expectOut != buffer.String() { t.Errorf("[%s]expect:n %vn but got:n %vn", name, test.expectOut, buffer.String()) } } } type badlyMarshaledSecret struct { api.Secret } func (a badlyMarshaledSecret) MarshalJSON() ([]byte, error) { return []byte(`{"apiVersion":"v1","Data":{"username":"--THIS IS NOT BASE64--"},"kind":"Secret"}`), nil } </ins>

Currently if one wants to view secrets stored in k8s they must follow the process described here: This involves copy&pasting. It seems to me that kubectl could provide a more helpful interface to base64 decrypt the users on behalf of the user saving them the trouble.
The kubelet does decode secrets before exposing them to the pod. Also the title doesn't really make sense. Secrets aren't encrypted so there's nothing to decrypt.
/kind feature
Yes, though this is low priority since we mostly don't want end users to view these once they get into the system.
Why can't that be an API permissions issue? I mean, if I really want to see it, I can just launch a pod to tell me.
I've update the issue title with what I understand as the request. To the best of my understanding this is just a request for us to avoid the need to pipe into base64 when inspecting secrets.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
I would like to try to make this work before it rots. cc Any guidance on a preferred implementation?
What do you think about adding support for functions in templates? This would allow someone to use the existing and a template that uses the function -- or a wide variety of other interesting functions as a side effect. I believe (but have not yet tested) that the following might be sufficient to enable the requested behaviour (plus the change to the dependencies list, of course):
Hi I suppose my thoughts on this issue have changed a little since I wrote it. The situation isn't quite as bad as I described as it is possible to decode a secret in one go without then need to copy and paste. i.e.

That said, that is a bit of a mouthful. Also, it might be nice to more easily 'work' with individual secrets and set them as well as get them. Should a user really need to pull down an entire group of secrets and edit a file just to be able to update a single password? For example something like this might be possible: What do people think? PS: Should this have been an issue on [kubectl] instead? PPS: I think some of the ugliness of this comes from configmaps being very similar to secrets except for the base64 encoding. Have there been any thoughts around bringing them into line? If memory serves, it might actually be nicer, due to YAML syntax-shenanigans, if configmaps were base64 encoded too.) [kubectl]:
Hi ... yes, I was thinking that it would be handy to be able to do something like: which is essentially the same as what you did without the overhead of the base64 process, or embed the information in a larger template like or in theory be able to use any of the other Sprig functions if someone found them useful. I think that having the Sprig methods available for generic formatting would just be useful on its own, really. On the topic of a or a , that seems pretty cool. There seem to be quite a lot of options for that I think people might expect to see mirrored in a if it were to exist. Thoughts?
It's also come to my attention that works great on MacOS but on Linux it's , so if someone, say, has a Helm chart where the notes tell the user to retrieve the secret data using one or the other of these, it won't work on the other one. would clearly be a simpler experience for the user than Hmm... thinking further: does does the idea conflict with the documentation in ? Perhaps we'd have to do something like: or similar?
/remove-lifecycle rotten
Small tip: you can use which works on both Linux and Mac.
Thank you
Update: I've looked at the huge amount of additional code that would be imported as a result of importing sprig and decided to go with a much smaller change instead, which is simply to implement a template function using . See PR with code and test.
please see the linked PR which addresses the original issue in a minimal fashion -- it simply provides a template function.

It has passed all the tests and needs review; if it's good enough to solve the original problem I'm happy to squash the commits and can hope to get it merged. As far as solving evolved description of the problem, I'm not sure where to start or if you really want me to go in that direction. Would you consider taking this change and splitting the additional work for being able to get / set the contents of an individual key in a secret into a separate issue? If you have any guidance on what you'd like to see with that I'd be happy to take a look, but it might be out of my current grasp.


Question: framework.ResourceEventHandlerFuncs{ AddFunc: jm.enqueueController, UpdateFunc: func(old, cur interface{}) { <del> job := cur.(*experimental.Job) for _, c := range job.Status.Conditions { if c.Type == experimental.JobComplete && c.Status == api.ConditionTrue { return } </del> <ins> if job := cur.(*experimental.Job); !isJobFinished(job) { jm.enqueueController(job) </ins> } <del> jm.enqueueController(cur) </del> }, DeleteFunc: jm.enqueueController, },

Ran a test executable with more than 20,000 times and no failures. Any hints as to how to get it to show up? I'm running 100 jobs at a time, so my system is stressed already.
I to am unable to reproduce this. I get similar errors if I reduce the timeout at the top of the file. I spent a bit looking at how to rewrite the tests without the timeout, but nothing jumped out at me. Looking at it though, it's really just a 500ms timeout for the test to complete. I'm just going to remove them. Go has a default timeout of 10 minutes. If it's going to pass, it will pass then. If not, Go will give you a nice traceback where it deadlocked.


Question: // exactly once. StartRequest(ctx context.Context, hashValue uint64, flowDistinguisher, fsName string, descr1, descr2 interface{}, queueNoteFn QueueNoteFn) (req Request, idle bool) <ins> // UpdateObservations makes sure any time-based statistics have // caught up with the current clock reading UpdateObservations() </ins> // Dump saves and returns the instant internal state of the queue-set. // Note that dumping process will stop the queue-set from proceeding // any requests.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Not observed yet, but deduced from analyzing the code. If a k/apiserver/pkg/util/flowcontrol/fairqueuing.QueueSet, whose metrics.TimedObserverPair is a pair of k/apiserver/pkg/util/flowcontrol/metrics.sampleAndWaterMarkHistograms, gets zero activity for a long while and then gets some activity, then the loop at will go through a lot of iterations, and do so with a lock held. What you expected to happen: No big hiccups in any scenario. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In reviewing I noticed an incorrect fake implementation of Interface. I looked at the comment on the Handle method, and noticed that it does not say when/how the is called. What you expected to happen: I expected the parameter to be as well documented as the others. How to reproduce it (as minimally and precisely as possible): Look at and see how it does not say when/whether/with-what-parameter-values is called. Anything else we need to know?: Environment: Kubernetes version (use ): current Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Any chance we could write a test for the interface which verifies correct plumbing?
/cc
: that should be do-able


Question: Phase: v1.PodFailed, ContainerStatuses: []v1.ContainerStatus{ { <del> Name: "main-container", </del> <ins> Name: "foo", }, { Name: "bar", </ins> State: v1.ContainerState{ Terminated: &v1.ContainerStateTerminated{ ExitCode: 2,

Since the last days there is an unusual rate of flakiness on kind jobs, but couldn't find a pattern ! While looking at one test failure and checking at the controller manager logs, there is a panic on the controller manager that makes the controller manager to restart and impacts on all the test that are running at that time. ` Lessons learned, we should not increase timeouts forever or these problems will go unnoticed, still, they are really hard to find and is not the first time that this happens, maybe Kubernetes is too good reconciling XD It will be great to have a tool that detect panics on the components /sig scheduling
/release blocker /priority important-soon /kind bug /cc
looks like the exit code matching doesn't handle a mix of terminated and unterminated containers, needs something like this:
big +1, a restart or panic detector on control plane components would be excellent
cc
this sounds like a nice project ^^^ but is better to check first with and , I see there are some scripts that parse and dump the logs here maybe we can add some thing to report them somehow?
/assign
oops, it's holiday in Poland /unassign /assign
/remove-sig scheduling
I can add the fix proposed by but the underlying problem is . I'll prepare a PR for each and see if we are ok merging both or just the hotfix.
commented on we'll merge the panic fix right away in either case
Fix and unit test is up


Question: // If the services are expected to keep running after test, they should not monitor the test process. e2es = services.NewE2EServices(*stopServices) gomega.Expect(e2es.Start()).To(gomega.Succeed(), "should be able to start node services.") <del> klog.Infof("Node services started.  Running tests...") </del> } else { klog.Infof("Running tests without starting services.") }

Which jobs are failing: Which test(s) are failing: The tests are failing in the . Since when has it been failing: 10/23/2020 Testgrid link: Reason for failure: It started failing at commit , but it's unknown if it's related to that. Anything else we need to know: We discussed in the Kubernetes SIG-Node CI subgroup and will start taking a look at it.
/assign
/sig node
Investigation notes: between successful and unsuccessful runs and found no notable differences. , and had a couple of failed health checks on initialization, which doesn't happen in the successful test runs, though there's only 2 seconds between starting readiness checks and an eventual successful check. config the same between runs looking through the kubelet logs for and , but I haven't found anything interesting yet. kubelet- failed health checks The below is a list of errors that occur in runs that succeeded and failed, which may be due to startup or specific testing scenarios. Bunch of these for numerous containers:
Both new and old test runs have a bunch of logs like this, implying that it's been having issues for a while: The failed one has this additional line: However, the doesn't have the same issues. I don't see the connection timeouts at all. I also don't see anything else in the kubelet logs that appears different between successful and failed runs. Unfortunately, I don't know enough about the infrastructure setup, but here are my current theories: about the infrastructure has gotten progressively more overloaded and now it's falling over. The tests run on 8 nodes versus 1, so they have more capacity there. Not totally buying this because why did it go from mostly successful to 100% failures rather than a mix for a bit. changed in the GCE configuration. Not sure how that is managed.
Diff there suggests three merges from k/k Only merge in test-infra is an image bump <img width="215" alt="image" src="https://user-"Interesting that a similar ci version and pull version of the job passes and the one ci version does not. pull: passing ci: failing ci: The failing ones do not have a node_e2e , but I don't see any of the expected log messages for failure of the services to start.
It is interesting that one passes and one does not.

Primary differences are the number of nodes, resource settings, and the fact that the conformance tests use Docker.
In addition to the services log missing, the overall system run logs are also missing. May be related to any decision we make for
Would we be able to kick off a Prow job with custom configuration? In other words, to rule in or rule out something in the image change, can we kick off a job with the latest everything else but use the older image? In reference to the decision on , are you suggesting we pause investigation and see if we decide to just move or change the conformance tests?
/triage accepted /priority critical-urgent
Catching up on this. qq: has anyone vetted ? Ise that pr went in around the time this job began to fail The beforesuite (AI: we should list out what it does), fails to communicate with nodes the mention of 6443 ports looks interesting here.
Good find. Definitely looks suspicious. I'll dig into that.
From : This comes from , which has some node-conformance specific test setup, so maybe something interesting there.
quick interruption for instructions on how to reproduce this one: I think this may work Focus and skip The FOCUS and SKIP arguments im getting from Image config file The job also uses the following config file Make sure to copy it and name it (the argument to IMAGECONFIG_FILE above). Additional arguments The test args to this job are I think these are the defaults already but just in case may be actually useful to add the following to the above command Miscellaneous If you run into trouble with the above commands try going through The info here came from
You'll also need to use the flag to make sure it runs in Docker, I believe.
ah yeah, good catch! Thank you. Don't know about your availability, but if you are all busy ima allocate sometime later today to dive deeper into the info we have here
I'm going to try to spend time of this in the afternoon (PDT). Specifically, I was going to try to verify that the logs from starting up the test suite look as expected and possibly try to run the case to reproduce.

My first suspicion was to have the server running in another port than 6443 (which Kubelet has configured), this is not the case: Only o.SecureServing.BindAddress is being set explicitly, but the port comes from o.NewServerRunOptions().
I think it's related to the movement of this line. I couldn't figure out how to link to the PR so I made a comment.
as you notice before there's no , and we should be seeing Before:
I can repro the issue now, and I think the line was moved upward to generate the token before starting the kubelet. Moving it up out of the if-statement may help partially, but in the conformance test the kubelet is manually started ahead of this code, so it may not matter.
Make sense. Do you mean moving this out of the conformance check?
Haha, yea, that message is deceptive... I think I'll make a PR to at least let it be more correct and give us more info.
for alerts to subscribers, PR created. Please comment.
looking green!
The output looks very strange, I have no idea why. Output has double-underscore-number for the different runs, instead of the image used. [__08] vs [cos-lts]
Ya, I noticed that. It's weird. Not sure how it's set, but you can see that the junit config names have a similar issue: -junitcos- -
Ok, found the root of the problem. Here's the command for the master suite: Note , which is not present in the conformance tests. but not sure how it eventually gets here.


Question: continue } klog.V(4).Infof("Volume: %s (pod UID %s) is marked as mounted and added into the actual state", volume.volumeName, volume.podName) <del> if volume.attachablePlugin != nil { </del> <ins> // If the volume has device to mount, we mark its device as mounted. if volume.attachablePlugin != nil || volume.blockVolumeMapper != nil { </ins> deviceMountPath, err := getDeviceMountPath(volume) if err != nil { klog.Errorf("Could not find device mount path for volume %s", volume.volumeName)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (block volmode)] disruptive[Disruptive] Should test that pv used in a pod that is force deleted while the kubelet is down cleans up when the kubelet returns. Since when has it been failing: 10/8/19 Testgrid link: https://k8s- Reason for failure: Introduced by but root cause is probably Anything else we need to know:
/assign
/milestone 1.17 /priority important-soon this is consistently failing in release-master-informing job could you please take a look?
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.17 /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.17


Question: rsCopy := existingNewRS.DeepCopy() // Set existing new replica set's annotation <del> annotationsUpdated := deploymentutil.SetNewReplicaSetAnnotations(d, rsCopy, newRevision, true) </del> <ins> annotationsUpdated := deploymentutil.SetNewReplicaSetAnnotations(d, rsCopy, newRevision, true, maxRevHistoryLengthInChars) </ins> minReadySecondsNeedsUpdate := rsCopy.Spec.MinReadySeconds != d.Spec.MinReadySeconds if annotationsUpdated || minReadySecondsNeedsUpdate { rsCopy.Spec.MinReadySeconds = d.Spec.MinReadySeconds

What happened: We created a Deployment and continuously updated an annotation on it periodically for a week. Eventually, we noticed errors in the controller-manager logs due to the annotation growing too large. Example annotations: controller-manager error: What you expected to happen: that do not change the revision number should not be to the revision history. revision history should be capped at a fixed number of elements, it should not be possible to have unbounded growth for Deployments that are frequently updated. How to reproduce it (as minimally and precisely as possible): a Deployment modify an annotation on it and note that the annotation gets the same number appended every time. Environment: Kubernetes version (use ):
/sig scalability Not 100% sure where this one fits, I couldn't determine which group owns the Deployment/ReplicaSet controllers, but this is a scalability problem
/sig apps
Agree with these, and if the limit is hit, the beginning of the history can be removed.
/assign


Question: klog.ErrorS(err, "can't open port, skipping this nodePort", "port", lp.String()) continue } <del> if lp.Protocol == "udp" { // TODO: We might have multiple services using the same port, and this will clear conntrack for all of them. // This is very low impact. The NodePort range is intentionally obscure, and unlikely to actually collide with real Services. // This only affects UDP connections, which are not common. // See issue: https://github.com/kubernetes/kubernetes/issues/49881 err := conntrack.ClearEntriesForPort(proxier.exec, lp.Port, isIPv6, v1.ProtocolUDP) if err != nil { klog.ErrorS(err, "Failed to clear udp conntrack", "port", lp.Port) } } </del> replacementPortsMap[lp] = socket } }

Which jobs are flaking: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are flaking: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service Testgrid link: https://k8s- Reason for failure: /sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
CRI-O CI is also hitting this failure. We have been unable to reproduce locally. our CI runs on aws
hmm, I only see 1 failure out of 4 and seems that the connectivity didn't work at all
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: JacobTanenbaum. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Saw the test flaking recently: Wondering if this is just a timeout issue.

Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.

I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.

After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?

3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.

so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.

Would love to help out in reproing this, I'm at a loss for good next step so any suggestions welcome!
that's the problem with these flakes, they are not easy to repro, and with this low rate of failures it may be caused for combination of running tests, environmental problems (cpu, memory, ...) , ... I always start for checking the current failures and try to find patterns
Let's take a look to one occurrence The tests fails because can't connect to the first pod If we check the code of the tests, we can see that it always uses the port as source port, we can use that to identify the nodePort assigned interesting thing here, we are source nating the pod ip, and we don't see any other conntrack entry The test verifies it has endpoints however, kube-proxy is using endpointlsices since this version but it has no entries so the iptables rules to forward the traffic are not there Seems we have to adapt this function in the e2e framework to use endpointslices cc: EDIT seems that this function is being used by all the network tests that can cause flakes if the endpoint is updated before the endpointslice (we switched recently kube-proxy to use the later)
Great catch. Ideally this function should validate if both Endpoint and appropriate EndpointSlice objects(s) exist.
Actually - let me take it and send a PR for it.
is out for review
Interesting failures here It fails to connect to the first pod
Hmm - I took a quick look at kube-proxy logs and these look fine to me....
I clicked too fast, sorry ... I was looking at the wrong service and I though I've found something
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.

just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job:


Question: }, { "ImportPath": "github.com/go-openapi/jsonpointer", <del> "Comment": "v0.18.0", </del> <ins> "Comment": "v0.19.0", </ins> "Rev": "ef5f0afec364d3b9396b7b77b43dbe26bf1f8004" }, { "ImportPath": "github.com/go-openapi/jsonreference", <del> "Comment": "v0.18.0", </del> <ins> "Comment": "v0.19.0", </ins> "Rev": "8483a886a90412cd6858df4ea3483dce9c8e35a3" }, {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Test has been failing since 3/24, at 9:19 Testgrid link: Reason for failure: Error during : Due to release v0.19.0 of and the morning of 3/23. /milestone v1.14 /kind failing-test /priority critical-urgent /cc
Adding sig-testing until someone claims it /sig testing
The file specifies both a version and a SHA commit. For the hash is the same for releases v0.17.0 through v0.19.0. The same goes for . The hash has been the same for releases v0.17.0 through v0.19.0. For both of these packages, the change from v0.18.0 (whats in ) to v0.19.0 seems to be the addition of a and files.
I think this is the presubmit job that corresponds to (which i'm hoping will be a valid solution) So just waiting to see if it works (runtime about an hour)
Forgot to mention that this also needs to be fixed in master applies the fix directly to 1.14 branch applies the fix to master
Thanks for monitoring it so closely and for preparing the fixes I think and a cherry-pick to the branch make sense. I'd rather hold onto merging them until after we've released later today (), especially for 1.14. My reasoning is that we've seen the tests stable with v0.18 of both dependencies for quite a while. Even if the changes are small, it might still be worth letting the CI run for a while if we update any dependencies. Because the job is very quick, I feel we can still decide to go for it later today if we decide otherwise; I'll hold back for now and let's speak at today's release meeting.
Hm, looking at it a bit more, there's no real change between the 2 releases: <img width="308" alt="Screen Shot 2019-03-25 at 10 37 11" src="https://user-"A of seems to have triggered the new releases, but none of the 2 dependencies seem to vendor it (directly or indirectly).

I'm happy with that going in, at least for . Let's still chat in a few hours about how to proceed.
godeps adds comments that correspond to the latest tag for the referenced sha. that means a new tag on the same sha broke our CI verification, so we need to update the generated Godep file to pick up the new comments. there are no code or content changes associated (since the SHA we're using is not changing), and the contents of will not change.
This is affecting all PR's against master as well:
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Repurposing this as umbrella issue for the godeps issue since it's affecting multiple branches
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//pkg/util/reflector/prometheus:go_default_library", "//pkg/util/workqueue/prometheus:go_default_library", "//pkg/version:go_default_library", <del> "//plugin/pkg/admission/admit:go_default_library", "//plugin/pkg/admission/alwayspullimages:go_default_library", "//plugin/pkg/admission/antiaffinity:go_default_library", "//plugin/pkg/admission/defaulttolerationseconds:go_default_library", "//plugin/pkg/admission/deny:go_default_library", "//plugin/pkg/admission/exec:go_default_library", "//plugin/pkg/admission/gc:go_default_library", "//plugin/pkg/admission/imagepolicy:go_default_library", "//plugin/pkg/admission/initialization:go_default_library", "//plugin/pkg/admission/initialresources:go_default_library", "//plugin/pkg/admission/limitranger:go_default_library", "//plugin/pkg/admission/namespace/autoprovision:go_default_library", "//plugin/pkg/admission/namespace/exists:go_default_library", "//plugin/pkg/admission/noderestriction:go_default_library", "//plugin/pkg/admission/persistentvolume/label:go_default_library", "//plugin/pkg/admission/podnodeselector:go_default_library", "//plugin/pkg/admission/podpreset:go_default_library", "//plugin/pkg/admission/podtolerationrestriction:go_default_library", "//plugin/pkg/admission/priority:go_default_library", "//plugin/pkg/admission/resourcequota:go_default_library", "//plugin/pkg/admission/security/podsecuritypolicy:go_default_library", "//plugin/pkg/admission/securitycontext/scdeny:go_default_library", "//plugin/pkg/admission/serviceaccount:go_default_library", "//plugin/pkg/admission/storageclass/setdefault:go_default_library", "//plugin/pkg/admission/webhook:go_default_library", </del> "//plugin/pkg/auth/authenticator/token/bootstrap:go_default_library", "//vendor/github.com/go-openapi/spec:go_default_library", "//vendor/github.com/golang/glog:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The current help message from kube-apiserver is wrong. It shows only the 'NamespaceLifecycle' plug-in is available, and the default value is . Users cannot know what are the available plug-ins, and they get confused where comes from. What you expected to happen: The correct output should be this: How to reproduce it (as minimally and precisely as possible): Run using master or at least 1.7.2. Anything else we need to know?: This is a regression introduced when refactoring code into generic server code. Environment: Kubernetes version (use ): 1.7.2+ ? Cloud provider or hardware configuration**: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Any Others: None
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, it is really exist. If no guys works on this, i will pick it up.
Thanks for confirming this. I'm already working on it. Will submit pr in minutes.
cool. :)


Question: false, }, { <ins> "all-created-subpath-directory-with-permissions", func(base string) error { return nil }, "test/directory", "test", maxPerm, false, }, { </ins> "directory-with-sgid", func(base string) error { return nil

What happened: Adding a VolumeMount to a container with a multi-part subpath results in only the leaf directory having 777 permissions. If you define e.g. then within the volume, and will have permissions, whereas will have permissions. This means that if you want to remove and eventually from within a pod, you will need to use a privileged pod. If the sub-directories already exist, their permissions are unchanged, and the order in which these subpaths are created appears to be the order in which they appear within the spec -- if we see first, then will be created with 755 permissions and will be created with 777 permissions, even if there's another volumeMount for further down. What you expected to happen: Sub-directories created to support a volumeMount should all have the same permissions as the leaf directory. How to reproduce it (as minimally and precisely as possible): the yaml below. The first example has volume mounts so that is created before ; the second example has volume mounts in reversed order, so is encountered before , which results in being unwritable. <details<summaryThis pod will complete</summary</details<details<summaryThis pod will fail</summary</detailsAnything else we need to know?: I believe this is the return of Environment: Reproduces on minikube and a (OpenShift) development cluster I have access to Kubernetes version (use ): minikube: OpenShift cluster: output: Cloud provider or hardware configuration: Minikube: v1.15.1; docker driver; hostpath volumes (all default) OpenShift cluster: AWS, EBS volumes OS (e.g: ): Minikube: Fedora 32 OpenShift cluster: RHEL 8.2 Kernel (e.g. ): Minikube: OpenShift cluster: Install tools: minikube v1.15.1 / Not sure Network plugin and version (if this is a network-related bug): N/A Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig storage
/assign
GitHub didn't allow me to assign the following users: mengjiao-liu. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: NodeName             string AuthorizationModes   []string <ins> // Mark the controller and api server pods as privileged as some cloud // controllers like openstack need escalated privileges under some conditions // example - loading a config drive to fetch node information PrivilegedPods bool </ins> Token    string TokenTTL *metav1.Duration

Dear All: I'm not sure if I'm right. When i use whith .I found controller-manager logs Someone can help me? Thanks a lot.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Please give more information about your set up cc
Sorry for report later. My openstack didn't support metadata service,but support config-drive service. More info:
Thanks Based on this, I think it's an openstack cloudprovider problem more than a kubeadm problem...
Thank you for help me. I will change the label.
/area cloudprovider
/remove-kind support
Should be fixed with you can specify a search order for metadata and config drive /close
Specifying a search order doesn't really solve the problem, especially if the metadata api isn't available which is what ZhenyangZhao said . I'm getting the same issue and since the config drive is my only option at the moment.
Like for this to be /open
can you please provide logs/versions for your problem in your environment? Thanks, Dims
Let me know if you need anything else Kubernetes version: 1.8.4 Docker version: 17.09.0-ce Linux: CentOS 7.4 Looking at the docker container log for the api-server (I don't have problems with the controller-manager):
and share the following:
Hi! Thank you for filing this issue. We appreciate your interest in Kubernetes and kubeadm. In order to make the kubernetes/kubernetes issue tracker more usable, we're routing issues to different repositories with different focuses. I think this issue belongs in the kubeadm repository. Please move this issue to the in order for the kubeadm team to be able to track this efficiently. If you disagree with my assessment of moving this issue, please speak up and/or reopen the issue. Finally, please try to reproduce the bug with the latest Kubernetes version, it might as well already have been fixed :) Thank you and have a good day!


Question: fp.netlinkHandle.EnsureDummyDevice(DefaultDummyDevice) activeBindAddrs := map[string]bool{"1.1.1.1": true, "2.2.2.2": true, "3.3.3.3": true, "4.4.4.4": true} <del> currentBindAddrs := []string{"1.1.1.1", "2.2.2.2", "3.3.3.3", "4.4.4.4", "5.5.5.5", "6.6.6.6"} </del> <ins> // This is ipv4-only so ipv6 addresses should be ignored currentBindAddrs := []string{"1.1.1.1", "2.2.2.2", "3.3.3.3", "4.4.4.4", "5.5.5.5", "6.6.6.6", "fd80::1:2:3", "fd80::1:2:4"} </ins> for i := range currentBindAddrs { fp.netlinkHandle.EnsureAddressBind(currentBindAddrs[i], DefaultDummyDevice) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area ipvs /area ipv6 What happened: with IPVS mode creates a interface for binding service IPs. kube-proxy periodically addresses it doesn't know about from the interface. It tries to unbind the IPv6 address over and over. The error is a little vague, but see below. The IP parsing or address deletion code may be a suspect. What you expected to happen: Either ignore the IPv6 address on the interface (its not hurting anything) or remove it correctly. How to reproduce it (as minimally and precisely as possible): Its perfectly natural for a new interface to have an inet6 address by default. If for some reason your interface doesn't have a similar address, you can add one to repro. Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.1 Cloud provider or hardware configuration: AWS, GCP, DigitalOcean, bare-metal, Azure OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. ): 4.14.67-coreos Install tools: Typhoon
The error message from is a little vague. Hop on a node to help out with its deletion. This gives a clue about where goes wrong. With IPv4 addresses, leaving off the is permitted (though with a scary warning). Having the wrong prefix size produces the same error. With more verbose logging, we see the mask was stripped before calling 's AddrDel, so the bug lies in kube-proxy's construction of . The mask is dropped . should be (, net.IPNet). However, it looks like much of kube-proxy ipvs deals in string IPs (e.g. 1.2.3.4, 1002:ab8::2:1) so propagating a or adding a mask could be a large change. One might also point to called as the source - it assumes a mask for IPv4 and a mask for IPv6.

Neither is neccessarily right for arbitrary addresses that might end up bound on a interface, but for IPv4 happens to work for Kubernetes IPv4 service IPs. for the IPv6 address does not. We could construct a netlink.IPNet without the helper, but by the time we reach , kube-proxy has already thrown away the mask information.
Any relation with this you think? Doesn't seem so since this is in and the other is ipvs targets...
Hm, I think adapting the sync'er for IPv6 might be a different fix. I was working on a fix for the IPv6 cleanup on the interface, lemme see if I can get that to a state it can be posted.
I can only reproduce this fault if I do; This will automatically assign a link-local generated address to the interface and the problem occurs. But the should not be up. By default if I start v1.12.0 it is down. Is some program, e.g network-manager, brings the interface up? Anyway I think it is possible to suspress auto-generated link-local ipv6 addresses. I will try to find the setting.
After creation; should fix this problem. I will try to get time to do a PR.


Question: } } <ins> func TestMakeMountsEtcHostsFile(t *testing.T) { testPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace:   "test-ns", Name:        "test-pod-name", Annotations: map[string]string{}, }, Spec: v1.PodSpec{ EnableServiceLinks: ptr.To(false), // to avoid errors with kl.makeEnvironmentVariables }, } testContainer := v1.Container{ Name:  "test-container", Image: "img", } tests := []struct { name        string podFn       func(pod *v1.Pod) containerFn func(container *v1.Container) podVolumes  kubecontainer.VolumeMap podIPs      []string want        bool // check if mounts the /etc/host file wantErr     bool }{ { name:   "pod with network", podIPs: []string{"192.168.0.1"}, want:   true, }, { name: "pod with network but no IPs", want: false, }, { name: "pod with host network", podFn: func(pod *v1.Pod) { pod.Spec.HostNetwork = true }, podIPs: []string{"192.168.0.1"}, want:   true, }, { name: "pod with host network but no IPs", podFn: func(pod *v1.Pod) { pod.Spec.HostNetwork = true }, want: true, }, { name: "pod with network already mounting the hosts file", containerFn: func(container *v1.Container) { container.VolumeMounts = []v1.VolumeMount{ { Name:      "etchosts", MountPath: etcHostsPath, }, } }, podVolumes: kubecontainer.VolumeMap{ "etchosts": kubecontainer.VolumeInfo{Mounter: &stubVolume{path: "/mnt/disk"}}, }, podIPs: []string{"192.168.0.1"}, want:   false, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { fhu := hostutil.NewFakeHostUtil(nil) fsp := &subpath.FakeSubpath{} pod := testPod.DeepCopy() if tt.podFn != nil { tt.podFn(pod) } container := testContainer.DeepCopy() if tt.containerFn != nil { tt.containerFn(container) } mounts, _, err := makeMounts(pod, t.TempDir(), container, "fakepodname", "fakedomain", tt.podIPs, tt.podVolumes, fhu, fsp, nil, false, nil) if err != nil { t.Fatalf("unexpected error: %v", err) } got := false for _, mount := range mounts { if mount.Name == "k8s-managed-etc-hosts" { got = true break } } if got != tt.want { t.Errorf("/etc/hosts file mounted: got = %v, want %v", got, tt.want) } }) } } </ins> func TestMakeBlockVolumes(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup()

When the node network is not yet connected, start kubelet and create static pod like this The created container file does not contain the configuration of hostAliases. The created container file should contain the configuration of hostAliases. kubelet service. static pod with configuration of hostAliases. network, like . kubelet service. created container file does not contain the configuration of hostAliases. No response <details</details<details</details<details</details<detailskubeadm </details<detailscontainerd 1.6.4 </details<details</details
/sig node
23 is EOL please check if this is happening in a supported release, latest supported is 1.25
You might want to set up a dependency on network from kubelet (assuming you're using systemd), so that it doesn't start until network is ready. Also, could you reproduce this on a later version (eg. 1.27)? /triage needs-information
The kubelet systemd configuration file already declares a network-online target dependency, but it doesn't seem to work. Looks like additional dependencies are required such as NetworkManager-wait-online.service. I think the best way is that kubelet is compatible with this scenario. Also, I also tested on 1.27 and the problem is still reproducible.
Is there any progress in dealing with this issue?
I haven't been working on this, not sure if anyone is actively looking into it.
did you try adding this dependency? Did it work?
NetworkManager-wait-online.service only waits for 60s. If the network is still abnormal after 60s, the problem will still occur.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network The kubelet wait for the PodIPs to be present to mount the etc hosts files however, it does not check if the pod is host network. If the Pod is using hostNetwork, then it seems the PodIPs are ignored so it seems legit to mount the aliases independently of PodIPs if the pod is using host network /assign /sig network /triage accepted
/remove-sig node


Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: TestDevicePluginReRegistrationProbeMode Testgrid link: Reason for failure: Anything else we need to know: Reproduce with /sig node
/assign
See
not sure this is actually fixed, I see a failure from today


Question: # Kubernetes Cluster Admin Guide: Cluster Components <ins> **Table of Contents** <!-- BEGIN MUNGE: GENERATED_TOC --> - [Kubernetes Cluster Admin Guide: Cluster Components](#kubernetes-cluster-admin-guide-cluster-components) - [Master Components](#master-components) - [kube-apiserver](#kube-apiserver) - [etcd](#etcd) - [kube-controller-manager](#kube-controller-manager) - [kube-scheduler](#kube-scheduler) - [addons](#addons) - [DNS](#dns) - [User interface](#user-interface) - [Container Resource Monitoring](#container-resource-monitoring) - [Cluster-level Logging](#cluster-level-logging) - [Node components](#node-components) - [kubelet](#kubelet) - [kube-proxy](#kube-proxy) - [docker](#docker) - [rkt](#rkt) - [monit](#monit) - [fluentd](#fluentd) <!-- END MUNGE: GENERATED_TOC --> </ins> This document outlines the various binary components that need to run to deliver a functioning Kubernetes cluster.

has TODOs for how to bring up cluster addons like monitoring and logging. We should fill these in. requested by user:


Question: export CONTEXT="ubuntu" export KUBE_SERVER="http://${KUBE_MASTER_IP}:8080" <del> source "${KUBE_ROOT}/cluster/common.sh" </del> # set kubernetes user and password load-or-gen-kube-basicauth

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: I was following to install a cluster but failed due to the following. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
The current workaround solution is remove in ./kube-
please have a look
thanks for the quick response
Just FYI, I think that you should fix both kube-up and kube-down
i have already move: source "${KUBE_ROOT}" to the head of ,is there something can't work?
Thanks, I did not have a try but just a reminder, just check code, yes, I think it works. Thanks
Seems the kube-up still does not work, filed another issue here
can you help take a look? Thanks!
/sig cluster-lifecycle
hi, I still see the issue but wondering since you have closed considering planned deprecation of the script. Does this issue still requires a fix or any decision was made not to work on it? Thanks!
Thanks , I think that we can close this now as it is not supported any more.
sure and thank you for the quick response and closing it !!


Question: func Register(registry k8smetrics.KubeRegistry) { registry.Register(healthcheck) registry.Register(healthchecksTotal) <ins> _ = k8smetrics.RegisterProcessStartTime(registry.Register) </ins> } func ResetHealthMetrics() {

/metrics/slis was for several components (apiserver, kube-controller-manager, kube-scheduler, kubelet, cloud-controller-manager) in This is to propose adding metric to these components for /metrics/slis to support cumulative metric Missing start time metric in metrics/slis will result in inaccurate cumulative metrics. Even though the start time metric is available in /metrics, it requires the metrics collector to scrape two endpoints at once and sync them to prepare the cumulative metric data.
/sig instrumentation
/assign cc /triage accepted
I just ran into an issue related to this change while adding scrapers to /metrics/slis in kube-prometheus The components /metrics and /metrics/slis endpoints are supposed to be scraped together, but at different intervals and this is really possible anymore since we the metric to the /metric/slis endpoint since now both endpoints are exposing the same metric which causes a conflict when scraped by Prometheus. Instead of exposing the metric inside of the endpoint, I think we should rather expect users to correlate the metric with the process start time metric from the endpoint. Because otherwise, we have to explicitly drop the metric at scrape time from the SLI endpoint in order to avoid conflicts.
We can't not include it, what if people opt only to scrape ?
That's a fair point, but do we really need to make use of the cumulative metric? Computing an error rate from it shouldn't require a start time, and even in the event of a restart, it should be transparent to the computation of the rate.
Yes, you need process-start-time in order to know when a process has restarted otherwise a cumulative metric could be (1) not cumulative going from 2 -1, (2) be cumulative but not measured as such if you measure 1 at process-start-time A, and then you measure 1 again at process-start-time B, but you ignore process-start-times.


Question: utilerrors "k8s.io/apimachinery/pkg/util/errors" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2" "k8s.io/kubernetes/pkg/kubelet/checkpointmanager" <ins> "k8s.io/kubernetes/pkg/kubelet/checkpointmanager/errors" </ins> kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/dockershim/libdocker" "k8s.io/kubernetes/pkg/kubelet/qos"

/kind bug are failing. seems like the likely cause. cc /assign
ping
seems one test still failing.


Question: podName  = "pod-" + string(uuid.NewUUID()) pod      = testPodWithROVolume(podName, source, volumePath) ) <ins> pod.Spec.NodeSelector = map[string]string{ "kubernetes.io/os": "windows", } pod = f.PodClient().CreateSync(pod) ginkgo.By("verifying that pod has the correct nodeSelector") framework.ExpectEqual(pod.Spec.NodeSelector["kubernetes.io/os"], "windows") </ins> <del> f.PodClient().CreateSync(pod) </del> cmd := []string{"cmd", "/c", "echo windows-volume-test", ">", filePath} _, stderr, _ := f.ExecCommandInContainerWithFullOutput(podName, containerName, cmd...) framework.ExpectEqual(stderr, "Access is denied.") } func doReadWriteReadOnlyTest(f *framework.Framework, source v1.VolumeSource, volumePath string) {

similar to PR en route
/sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details