Question: return clientset.Core().Namespaces().Create(ns) } <del> func createService(cmdOut io.Writer, clientset client.Interface, namespace, svcName, federationName, apiserverAdvertiseAddress string, apiserverServiceType v1.ServiceType, dryRun bool) (*api.Service, []string, []string, error) { </del> <ins> func createService(cmdOut io.Writer, clientset client.Interface, namespace, svcName, federationName, apiserverAdvertiseAddress string, apiserverPort *int32, apiserverServiceType v1.ServiceType, dryRun bool) (*api.Service, []string, []string, error) { port := api.ServicePort{ Name:       "https", Protocol:   "TCP", Port:       443, TargetPort: intstr.FromString(apiServerSecurePortName), } if apiserverServiceType == v1.ServiceTypeNodePort && apiserverPort != nil { port.NodePort = *apiserverPort } </ins> svc := &api.Service{ ObjectMeta: metav1.ObjectMeta{ Name:        svcName,

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.6.2 Environment: Cloud provider or hardware configuration: AWS using Juju OS (e.g. from /etc/os-release): Ubuntu Xenial Kernel (e.g. ): 4.4.0-59-generic Install tools: Juju Others: What happened: kubefed init entered a infinite loop health checking the federation api service What you expected to happen: kubefed to exit with a success code How to reproduce it (as minimally and precisely as possible): Anything else we need to know: When doing a kubefed init on a cluster setup using VMs on aws (using Juju in my case) the worker nodes have two IPs: one that is visible to the internet and one internal visible only to VMs within aws. In my case for example the VM on aws hosting the kubernetes-worker node had external IP in the 54.174.73.1/24 subnet and it also had an internal IP in the 172.31.60.1/24 subnet. Using a client machine outside aws to bootstrap federation: I got the kubefed client stuck on doing a health check against the clouds internal subnet https://172.31.60.XXX:/healthz . To address this I can use the flag and pass the external IP of any kubernetes-worker node.

With this flag kubeconfig file gets updated with the right endpoint for the federation-apiserver. However, apart from the I should also be able to specify the desired port the service will be listening on. I have to make sure that port is opened on the aws routing/firewall prior to issuing . In the example above, kubefed init will try to reach https://54.174.73.XXX:/healthz, port is randomly assigned and is not reachable. I should be able to first open the port on aws and then do a with and port pinned to .
/cc
/assign
is there a reason why you are not using the default load balanced service on AWS?
I am using Juju to stay cloud agnostic. AWS just happened to be where I deployed this time. Juju just grabs machines (virtual or physical) from wherever you have resources from. In effect your cluster is more like an on-prem than anything else. How would the default load balanced service work?
got it.


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/credentialprovider" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> docker "github.com/fsouza/go-dockerclient" )

Is this to follow pointers?
Context: The follows pointers when printing. Unlike JSON marshaling/unmarshaling it also handles private fields.

I am experiencing some weird issues when enabling HTTP liveness probes for some containers. I configure the following for all HTTP-serving containers (this is part of a template used by a replication controller with a single replica): When this is configured, the following issues start to occur: One pod is already running without problems. When I add another replication controller which creates a new pod (also with the liveness probe set), the first pod (the one that was already running) is restarted. This happens even when the two pods are in different namespaces. But when a pod's HTTP server fails completely (connection to the port is refused), the liveness probe will not trigger and the pod will not be restarted. This behavior occurs consistently when I start with no replication controllers and then add them one by one. When I disable the liveness probes on the affected pods/templates, they are no longer restarted in this manner.
You say you're starting two replication controllers-- can you verify that their label selectors do not intersect?
Yes, each replication controller has a selector over a single label in the form of so the labels do not overlap. Also, this problem occurs even when the apps are in different namespaces and goes away when the liveness probe is removed.
Thanks-- last question-- Do the pods have to be scheduled on the same node for this to occur? Your second point I believe is a problem here: we treat an error during HTTP GET as healthy for some inexplicable reason. I can't really explain your first observation, unless perhaps the pods and containers have the same names, but in different namespaces AND kubelet has a namespace handling bug?
cc
"Unknown" != "Healthy" I think it's an open question about whether we want to treat a GET failure as "Unhealthy" we certainly could... --brendan , Daniel Smith wrote:
Giving this to Saad. I think it's pretty clear that if kubelet can't do an HTTP GET, then the pod isn't healthy-- they're on the same host, HTTP get can't fail for network reasons. Our internal system makes this assumption. (I broke it once!)
tl;dr... When a second replication controller is the hash of the container of the first replication controller changes causing it to be killed and recreated. The hash changes because the LivenessProbe pointer changes. Need to figure out why that happens.

Spent time trying to understand why creating two replication controllers, both with http liveness probes, causes the container from the first replication controller to be killed and then recreated when the second replication controller is created. It turns out, as soon as the second replication controller is created, the hash of the container from the first replication controller changes and thus the container is killed: Let's see how the expected and actual hashes are computed and how they end up differing. The actual hash is pulled from the docker container name (dockerContainers.FindPodContainer): The expected hash is created from the object. Specifically, this code (): Expected hash before second replication controller is : Expected hash after second replication controller is : The difference between the container before and after is: So the hash changes because the LivenessProbe pointer in the object changes. Need to figure out why this happens.
Great detective work-- I think it's clear that we shouldn't be including pointers in our hash, so we need to revisit that and explicitly hash only the things that we care about.
Adding second replication controller changes the pointer of LivenessProbe in container config is another mystery. We need to make sure not randomly update other fields too.
Dug in to this some more: confirmed that the value of the original container's LivenessProbe doesn't actually change when the second replication controller is created--only the pointer to the LivenessProbe object changes. It turns out that the instance of the  object being used to compute the expected hash is not guaranteed to be the same instance for the lifespan of the actual docker container. In fact, every time there is an etcd update (e.g. a new pod is created) a new boundpods object is created and although the values of the existing pod/container container objects don't change, because new instances are created the pointers do change (by design). So the fix for this is to ensure that the expected hash is computed only on container values not pointers. One way to ensure this would be to make sure the container object specifies no pointer fields. However, LivenessProbe and Lifecycle, since they are optional, must be able to be set to . Thus, pointer fields can't simply be converted to value fields. Another approach would be to only hash some subset of the values (maybe the name and image) of the container.

However, containers should be destroyed/recreated (meaning the hash must change) if the actual values of any of the container fields changes (e.g. the cpu, memory, volume mounts, liveness probe path, etc.), not just the container name/image. Thus, pointer fields can't simply be disregarded for the hash computation. Instead the fix must modify the expected hash to be computed on the complete deep values of the entire container object (including the actual LivenessProbe values, instead of the pointer). The easiest way to do this is to marshal the container object to json and compute the hash from the resulting byte array.


Question: ResourceVersion:   "105", }, verifyBackingStore: func(t *testing.T, s *dummyStorage) { <del> require.NotEqual(t, 0, s.requestWatchProgressCounter, "expected store.RequestWatchProgressCounter to be > 0. It looks like watch progress wasn't requested!") </del> <ins> require.NotEqual(t, 0, s.getRequestWatchProgressCounter(), "expected store.RequestWatchProgressCounter to be > 0. It looks like watch progress wasn't requested!") </ins> }, },

Internal CI for ppc64le arch : https://prow.ppc64le- cacher Failure is observed on Mar 18th No response No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This test is flaking with data race as is getting accessed for field without obtaining locks in I am thinking to raise a PR to add necessary locks for different /cc
/assign


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/net" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> clientset "k8s.io/client-go/kubernetes" <ins> "k8s.io/client-go/transport" </ins> "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2erc "k8s.io/kubernetes/test/e2e/framework/rc"

[x] two weeks soak start date : testgrid-link [x] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining ProxyWithPath endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a pod with a static label using the agnhost image Confirm that the pod is in the running phase Iterate through a list of http methods and check the response from the porter app Confirm that each response code is 200 OK If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance Where/How does PATCH/HEAD/OPTIONS requests for the following endpoints line up with the API reference? connectCoreV1PatchNamespacedPodProxyWithPath connectCoreV1HeadNamespacedPodProxyWithPath connectCoreV1OptionsNamespacedPodProxyWithPath Pod v1 core: Proxy Operations the following endpoints are listed as valid endpoints; connectCoreV1HeadNamespacedPodProxyWithPath connectCoreV1OptionsNamespacedPodProxyWithPath These requests are getting logged as a verb. Line : api/openapi-spec Line : api/openapi-spec Looking at there doesn't look to be a mapping for the above methods. According to this APIsnoop query, there are still some remaining ProxyWithPath endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a pod with a static label using the agnhost image Confirm that the pod is in the running phase Create a service and link it to the pod Iterate through a list of http methods and check the response from the porter app Confirm that each response code is 200 OK If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance.

The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/re-open
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> # A project wanting to generate openapi code for vendored # k8s.io/kubernetes will need to set the following variables in # //build/openapi.bzl in their project and customize the go prefix: # # openapi_go_prefix = "k8s.io/myproject/" # openapi_vendor_prefix = "vendor/k8s.io/kubernetes/" openapi_go_prefix = "k8s.io/kubernetes/" openapi_vendor_prefix = "" </ins>

As part of moving federation out of tree (), the federation build will need to generate ` for itself and vendored kubernetes. The following steps are required: [x] Discover targets for generation () [x] Update the file's bazel rules to configurably enable a build for vendored kube () [x] Support generation of the file to a vendored path () [x] Update kubernetes to use latest kazel and kube-openapi () /sig multicluster testing /area federation /assign cc:
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: } csiMounter := mounter.(*csiMountMgr) <ins> mountPath := filepath.Dir(csiMounter.GetPath()) err = prepareVolumeInfoFile(mountPath, plug, tc.originSpec.Name(), csiMounter.volumeID, testDriver, string(csiMounter.volumeLifecycleMode)) if err != nil { t.Fatalf("failed to save fake volume info file: %s", err) } </ins> // rebuild spec spec, err := plug.ConstructVolumeSpec("test-pv", filepath.Dir(csiMounter.GetPath())) if err != nil {

Instantiating the interface causes the creation of pod local volume directory and json data file and not - This causes a problem where is failed (and as a result never gets called) then those pod local volume directories will never be cleaned up. /sig storage cc
Looks like is the PR that introduced this change and moved data file creation calls to .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/repen /remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
/remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue.

<detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /lifecycle frozen
Reopened this issue. <detailsIn response to : /reopen /lifecycle frozen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment.

<detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err != nil { return nil, statusError(logger, err) } <del> s.availableOnNodes[index] = nodeSelector </del> <ins> s.informationsForClaim[index].availableOnNode = nodeSelector } if claim.Status.Allocation == nil && claim.Spec.AllocationMode == resourcev1alpha2.AllocationModeWaitForFirstConsumer { // The ResourceClass might have a node filter. This is // useful for trimming the initial set of potential // nodes before we ask the driver(s) for information // about the specific pod. class, err := pl.classLister.Get(claim.Spec.ResourceClassName) if err != nil { // If the class does not exist, then allocation cannot proceed. return nil, statusError(logger, fmt.Errorf("look up resource class: %v", err)) } if class.SuitableNodes != nil { selector, err := nodeaffinity.NewNodeSelector(class.SuitableNodes) if err != nil { return nil, statusError(logger, err) } s.informationsForClaim[index].availableOnNode = selector } // Now we need information from drivers. schedulingCtx, err := s.initializePodSchedulingContexts(ctx, pod, pl.podSchedulingContextLister) if err != nil { return nil, statusError(logger, err) } s.informationsForClaim[index].status = statusForClaim(schedulingCtx, pod.Spec.ResourceClaims[index].Name) </ins> } }

The node selectors gets parsed by in for each node. That work can be done once in . Better performance.
/sig node /sig scheduling /triage accepted /priority backlog
I can try this one. /assign


Question: # HELP apiserver_encryption_config_controller_automatic_reload_success_total [ALPHA] Total number of successful automatic reloads of encryption configuration split by apiserver identity. # TYPE apiserver_encryption_config_controller_automatic_reload_success_total counter apiserver_encryption_config_controller_automatic_reload_success_total{apiserver_id_hash="sha256:cd8a60cec6134082e9f37e7a4146b4bc14a0bf8a863237c36ec8fdb658c3e027"} 1 <ins> # HELP apiserver_encryption_config_controller_automatic_reloads_total [ALPHA] Total number of reload successes and failures of encryption configuration split by apiserver identity. # TYPE apiserver_encryption_config_controller_automatic_reloads_total counter apiserver_encryption_config_controller_automatic_reloads_total{apiserver_id_hash="sha256:cd8a60cec6134082e9f37e7a4146b4bc14a0bf8a863237c36ec8fdb658c3e027",status="success"} 1 </ins> ` const expectedFailureMetricValue = ` # HELP apiserver_encryption_config_controller_automatic_reload_failures_total [ALPHA] Total number of failed automatic reloads of encryption configuration split by apiserver identity. # TYPE apiserver_encryption_config_controller_automatic_reload_failures_total counter apiserver_encryption_config_controller_automatic_reload_failures_total{apiserver_id_hash="sha256:cd8a60cec6134082e9f37e7a4146b4bc14a0bf8a863237c36ec8fdb658c3e027"} 1 <ins> # HELP apiserver_encryption_config_controller_automatic_reloads_total [ALPHA] Total number of reload successes and failures of encryption configuration split by apiserver identity. # TYPE apiserver_encryption_config_controller_automatic_reloads_total counter apiserver_encryption_config_controller_automatic_reloads_total{apiserver_id_hash="sha256:cd8a60cec6134082e9f37e7a4146b4bc14a0bf8a863237c36ec8fdb658c3e027",status="failure"} 1 </ins> ` tests := []struct {

Change encryption config controller reload metrics to be consistent with other places (Authn, Authz) xref: Combine and into a single metric with with an enum value of and . /sig auth /triage accepted
cc
We probably should keep the existing metrics for some time to allow folks to migrate to the new one.
Ah I was thinking about this as well! Looking at "may be deleted at any time", I was leaning towards just removing it. If the common approach is to keep it for a release before removing, we could do that. I believe we can set a for the 2 metrics we want to remove.
I don't have a good sense of metrics stability because they tend to stay alpha forever and folks don't go back to update them. Since the associated feature set is GA, I think we should be conservative with the changes we make. Perhaps SIG instrumentation folks can provide some guidance here?
As part of the PR, I was going to ask when is the right time to graduate these metrics do you have thoughts on to take while deleting an alpha metric metrics should be graduated from alpha to beta and stable?


Question: if exist == false { t.Errorf("waitForPathToExist: could not find path %s", devicePath[1]) } <del> exist = waitForPathToExistInternal(devicePath[1], 1, "default", os.Stat, fakeFilepathGlob) </del> <ins> exist = waitForPathToExistInternal(devicePath[1], 1, "tcp", os.Stat, fakeFilepathGlob) </ins> if exist != false { t.Errorf("waitForPathToExist: wrong code path called for %s", devicePath[1]) }

open-iscsi software initiators are currently broken for non- interfaces due to the fact that the devicePath (in ) is assumed to use the prefixing scheme for all interfaces not named . This causes waitForPathToExist to timeout after 10 seconds since the device path that it is waiting for is not the - prefixed path that is created, resulting in host logging such as: or K8S event logs such as: open-iscsi support for hardware transports was in the following commit: In this commit, software iSCSI support was assumed to be done through only the use of the default interface. According to the documentation at (Section 5.1.1) both software and offload/hardware interfaces are intended to be configured using named interface specifications and is only a backwards-compatibility vehicle. Configuration of cannot be done via the use of as this approach is specifically forbidden in the documentation: Leaving this here to help others associate various failure modes with this issue and possible resolution: In my case I am just learning and this is my first iSCSI deployment. My initial error was that I was receiving an message at discovery-time even though I had used to define a name. This approach failed because I didn't realize that iscsid needed to be bounced before it would pick up changes to that file. I had defined a custom specification and was using that to accomplish the task of setting my initiator name. I was able to unwind this mess and return to the use of once I read the code and understood what it was attempting to do.
i'm marking this as a feature request against the iscsi volume driver. RFE: support iscsi interfaces named something besides 'default'.
In theory the fix should be easy, just move the path check from iface name to transport_name. I will take a look asap, unfortunately I am travelling till monday
Sorry about the previous comment, I was logged in to the wrong account. In theory the fix should be easy, just move the path check from iface name to transport_name. I will take a look asap, unfortunately I am travelling till monday. can you share your iface file ? It will help me validate
I no longer have the file precisely as it was. However, I did have some version of what I had used committed into my VCS. While not 100% sure this is what I was using while encountering the failure it does looks to be about what I'd expect:


Question: kube::test::get_object_assert 'pods a -n nsb' "{{${id_field:?}}}" 'a' # apply b with namespace kubectl apply --namespace nsb --prune -l prune-group=true -f hack/testdata/prune/b.yaml "${kube_flags[@]:?}" <del> # check right pod exists kube::test::get_object_assert 'pods b -n nsb' "{{${id_field:?}}}" 'b' # check wrong pod doesn't exist output_message=$(! kubectl get pods a -n nsb 2>&1 "${kube_flags[@]:?}") kube::test::if_has_string "${output_message}" 'pods "a" not found' </del> <ins> # check right pod exists and wrong pod doesn't exist kube::test::wait_object_assert 'pods -n nsb' "{{range.items}}{{${id_field:?}}}:{{end}}" 'b:' </ins> # cleanup kubectl delete pods b -n nsb

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig cli /cc /milestone v1.20 /prority important-soon
after creating the CRD, before applying the CR, kubectl should wait until includes :
/assign
/assign is this issue related to ?
no, this is simply kubectl trying to apply a CR before the server has made the API available at all
Got it! and I have filed to address this issue.
Still happening:
still happening: looks like the root cause is not related to CRD: I think the message Error from server (NotFound): "myobj" not found is expected, as the CR was not actually created: /reopen
Reopened this issue. <detailsIn response to : still happening: I think the message Error from server (NotFound): "myobj" not found is expected, as the CR was not actually created: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yes, the actual error is later: Line 177 succeeds, and the next line fails:


Question: // to be served from cache and potentially be delayed relative to // etcd contents. Reflector framework will catch up via Watch() eventually. options := metav1.ListOptions{ResourceVersion: "0"} <del> r.metrics.numberOfLists.Inc() start := r.clock.Now() </del> if err := func() error { initTrace := trace.New("Reflector " + r.name + " ListAndWatch")

What happened: After upgrading to kubernetes 1.12.5 we observe failing nodes, that are caused by kubelet eating all over the memory after some time. ! I use image to run kubelet on 102 clusters and since a week we see some nodes leaking memory, caused by kubelet. I investigated some of these kubelets with strace and pprof. With 3s of running strace I saw = 50 openat() calls to the same file from the same threadid (pid) from kubelet: If I do pprof kubelet it shows clientgo metrics and compress is taking over most of the compute time. Memory profile png: ! The reflector metrics seem to be the problem: What you expected to happen: I expect that kubelet does not need so much memory How to reproduce it (as minimally and precisely as possible): I don't know Anything else we need to know?: One of the affected clusters has only 120 Pods and 3 Pods are in CrashLoopBackOff state, one pod since 6 days on an affected node which was investigated. Environment: Kubernetes version (use ): v1.12.5 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Linux ip-172-31-10- 4.14.63-coreos SMP Wed Aug 15 22:26:16 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux Install tools: custom
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it might be a side effect of , but I couldn't find any numbers in there. The comments mention an offline discussion, was this brought up?
What happens (I only investigated it very briefly) is that kubelet creates a bunch of reflectors for every pod — secret mounts, configmap mounts, token mounts etc. Each reflector registers a number of histograms, gauges, etc. and even when the reflector is stopped and removed the metrics are never removed.

After a few days on a busy cluster you get millions of metrics bloating up kubelet and everything else using the reflectors.
This was broken since v1.12.0, so probably unrelated to .
correct that PR is only in v1.14.0-alpha.2 v1.14.0-alpha.1 and master
For everyone that finds this issue and needs a patch to disable the reflector metrics:
Do you have same metrics after disabling ReflectorMetricsProvider?
no, it drops the reflector metrics
is this indirectly caused by switching to watch-based managers?
Yeah - switching to watch resulted in more extensive using of a reflector. It seems that those metrics are directly coming from reflector (I wasn't aware of those). So it's highly probably that this change cause it. I think that we don't really need those metrics, so if we could switch them off in Kubelets, that should solve the problem.
is sent out for review to fix that. We should probably cherrypick it to previous releases if we agree that's the right way to solve it.
Actually, I realized that I don't fully understand the problem. It should be a problem in default Kubernetes (what we build from this repo), because those metrics are not switched on for kubelet: metrics factory is defaulted to noop the prometheus metrics need to be explicitly linked and initialized (as they are in apiserver e.g.: )
Wouldn't it be better to rewrite the reflector metrics so they're aggregated in a better way rather than relying on people not accidentally enabling them? Or at least put a huge warning on top saying "this will leak memory like crazy" just so the same situation doesn't repeat in other code using client-go?
- that's a separate issue whether the metrics are reasonable or not (and node sig-node related). One thing that I would like to confirm: do you customize k8s yourself? I would like to ensure that this issue you is in your setup and not in upstream k8s.
Our production configuration is this one: Kubelet config: We run with an image called , but it's just a mirror of . So upstream Kubernetes.
I try to figure out an issue why these metrics are even there and I found issues from the past showing leaks already in older versions: Sounds for, that we need a postmortem to make this not happening again.

I don't find any issue in kubernetes nor in client-go, that could reasonable explain why these were introduced in the first place. I guess I just did not found it, and I hope someone can highlight where the decision came from.
I am running Kubernetes v1.12.3 and I don't see reflector metrics being used by kubelet ( shows nothing). FWIW, the cluster is created using kops. Any idea, how are reflector metrics enabled?
and myself will also look into this as well and see what we can find.
Thanks
The issue only exists if hyperkube is used when running kubelet, as hyperkube has imported the package for apiserver and controller-manager. If kubelet binary is used, then there're no reflector metrics (this is same with your findings). Since the metrics is not exposed by kubelet cmd, what do you think of disabling it for kubelet subcommand in hyperkube? I can't reproduce the issue (with hyperkube used) on my own cluster: ! Since both apiserver and controller-manager have also imported the package, have you also observed same issues for them?
+1
Yes, we observe the same thing for apiserver and controller-manager as well.
I would also suggest not using imports for side effects because this usually creates hard to debug and non-obvious problems like this.
So you mean disable the reflector metrics totally?
No, I suggest changing the code so that the end users (kubelet, apiserver, etc) would have to explicitly enable the metrics by calling a function instead of having this happen as a side effect of an import statement.
LGTM. WDYT? We should probably do this for both kubelet and control plane services (apiserver and controller-manager).
I'm fine with that.
+1
+1
Why wouldn't we want to just actually disable this metric wholesale? Enabling this flag would basically be the same thing as saying enable a memory leak, no?
/assign
Han points out that, since the metric has a random suffix each time, it's not very useful for monitoring anyway.
/sig api-machinery


Question: } } <ins> func toRealServerKey(rs *utilipvs.RealServer) *realServerKey { return &realServerKey{ Address: rs.Address, Port:    rs.Port, } } </ins> //AddVirtualServer is a fake implementation, it simply adds the VirtualServer into the cache store. func (f *FakeIPVS) AddVirtualServer(serv *utilipvs.VirtualServer) error { if serv == nil {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: IPVS real server can't be right deleted in FakeIPVS for ipvs testing. What you expected to happen: When delete ipvs real server, we can't give the weight information, so we can't use deep equal in .
/sig network /area ipvs
/kind bug


Question: "//vendor/github.com/pkg/errors:go_default_library", "//vendor/k8s.io/klog:go_default_library", "//vendor/k8s.io/utils/exec:go_default_library", <del> "//vendor/sigs.k8s.io/yaml:go_default_library", </del> ], )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add an external MetaFactory that allows for serializing multiple YAML (or JSON) files in kubeadm. The logic of marshalling YAML should be factored out from to . Why is this needed: This will allow processing component configs containing multiple documents per file. Additionally, an out-of-tree library would allow sharing this functionality with other components.
I will try work on this as carry-on from /sig cluster-lifecycle /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: - name: number type: scalar: numeric <ins> elementRelationship: atomic </ins> - name: io.k8s.api.networking.v1alpha1.IPAddress map: fields:

I created an Ingress resource with a port number manually, and then tried to use an Operator with server-side apply to update the resource from a port number to a port name. This failed with the error message "cannot set both port name & port number". I expected the port number to be cleared as the operator would take over the ownership of the ServiceBackendPort. To simulate the behavior in the operator in a most simple example I use : ingress- ingress- This will print: It seems to me that the merge type for the struct ServiceBackendPort should to be set to "atomic" , so that the Operator updating the port should take over the whole struct, and two applications shouldn't update single fields in it. Red Hat OpenShift on AWS OS version of the client: NAME="Fedora Linux" VERSION="40 (Workstation Edition)" Linux 6.9.6-200.fc40.x8664 SMP PREEMPTDYNAMIC Fri Jun 21 15:48:21 UTC 2024 x86_64 GNU/Linux
/sig network
So, on one hand you made a problem when you used two different owners to manage one aspect of the object. On the other hand, I think you're right that it SHOULD work. Probably. is not a super common tag, and ingress way predates it, so no surprise it isn't used. I ACK the issue and reproduced it. I ran update-codegen and built a KinD cluster from HEAD.
this may be something you want to look at for DRA API - I didn't even realize this was a thing until today :)
The API uses for , which is very similar. So using it for LGTM. SSA will auto-migrate existing resources using once the change is released.
/triage accepted
"this" being "add "? I can see where this might be useful, probably on all one-of structs and perhaps on those containing a discriminator because otherwise SSA leaves fields set which should be cleared when some other owner takes over. Is this important enough to put into I don't see a big need for SSA in general and even less for SSA with different owners at the moment - this will change once we do more with status.
No, not urgent. This is the first time it has come up in my awareness ever. But now that I know about it, I could see how it applies to dra.

Thank to everyone involved here to resolve this! This will make things simpler in the Keycloak operator once it is released.
Thanks for filing a good (and easy to fix :) bug. Easy repro means we can jump on it.


Question: } v1beta1 := fake.NewSimpleClientset(&networkingv1beta1.Ingress{ ObjectMeta: metav1.ObjectMeta{ <del> Name:      "bar", </del> <ins> Name: "bar", Labels: map[string]string{ "id1": "app1", "id2": "app2", }, </ins> Namespace: "foo", }, Spec: networkingv1beta1.IngressSpec{

When command 'kubectl describe ing' Name: test-ing Namespace: practice Address: 10.10.01.10 Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found) TLS: test terminates Rules: Host Path Backends ---- ---- -------- / test-svc:80 (<none) Annotations: nginx Events: <noneResult has no information about labels. But Pod, Deployment, Service resources has label information like below. Name: test-room-test-mjsbz Namespace: test Priority: 0 Node: aks-test-test-test/22.22.222. Start Time: Fri, 09 Jul 2021 05:45:26 +0000 Labels: app=test pod-template-hash=test role=mediatest Why ingress has no information about labels? Actually when delete ingresss with labels is working correctly. Kubernetes version (use ): 1.18.14 Cloud provider or hardware configuration: AKS (Azure) OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig
/sig
/sig cli
/assign
/assign
/triage accepted /remove-kind bug /kind feature


Question: # Optional: customize runtime config RUNTIME_CONFIG="${KUBE_RUNTIME_CONFIG:-}" <ins> if [[ "${KUBE_FEATURE_GATES:-}" == "AllAlpha=true" ]]; then RUNTIME_CONFIG="${KUBE_RUNTIME_CONFIG:-api/all=true}" fi </ins> # Optional: set feature gates FEATURE_GATES="${KUBE_FEATURE_GATES:-ExperimentalCriticalPodAnnotation=true}"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Probably caused by <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
https://k8s- /assign
I understand that moved crd creation logic to e2e test's k8s cluster deployment code. On the other hand, I tested it with manually deployed k8s cluster. So, in my case, I guess that in addition to enable and feature gates enabled, I need to create crd manually before tests, like below: I could make two of three SkipAttach tests pass in my environment by above way. For "non-attachable volume does not need VolumeAttachment" case, I'm still looking into it. (It doesn't seem to skip creating volumeAttachment, properly)
I found a typo in 's argument. While hostpath driver's connects driver name and unique name without "-", it connects them with "-". This makes "non-attachable" test fail, due to the mismatch between volume name in crd and provisioner name, as a result, creating volumeAttachment isn't skipped. I will create a PR to fix it. For above bug, is unrelated. It just changed the way in how to test in manually deployed k8s cluster as I mentioned in above comment. So, there should be no problem for the PR, at least for the CI testing if it works well in CI environments.
I can verify that the CRDs are not getting installed on the alpha cluster. Investigating why.
/reopen since I see the Alpha tests failing
/milestone v1.13 /priority important-soon /kind failing-tests
/kind failing-test
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is fully resolved. There are still failures on https://k8s- /reopen
Reopened this issue. <detailsIn response to : I don't think this is fully resolved. There are still failures on https://k8s- Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you help take a look at these failures? It's testing CSI filesystem volume mode. I think it is unrelated to the crd installation issue that Saad fixed.
I tested with AllAlpha flag and manually created csi crds and confirmed that filesystem volume mode test passed in my environment and also other tests passed there. If I delete csi crds, other tests also fail, so I agree that the issue that Saad fixed is not related. However, above also means that this issue can't be reproduced in my environment.
I1109 14:56:01.141] Nov 9 14:56:01.131: INFO: At 2018-11-09 14:50:54 +0000 UTC - event for security-context--e42e-11e8-9b95-: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc--e42e-11e8-880b-" : node "bootstrap-e2e-minion-group-3wtg" has no NodeID annotation
Hostpath also seems to be failing because of driver registration issues? Not sure why other alpha tests are passing if that's the case
This test is the only test that uses  to create pod with CSI driver volume and expects the result to be success. One of the pod parameters specified there might affect this failure.
Those 2 failing tests are newly introduced in and started to fail from the beginning. Is that PR blocking 1.13? If not, can we revert and fix later? I have recently set up optional presubmit tests for alpha features at , which will pick up those 2 tests. Currently this failure is blocking
apart from the 2 failing tests, I see which addressed with is also flaky. These failures are now blocking PRs as mentioned above. As we enter code slush today, its very critical to have the PRs get in sooner than later. Can we plz revert the feature and the tests until they are fully resolved and ready to go. Thanks
I prefer disabling the specific tests instead of reverting the entire pr. We will lose important signal for csi
That works for me.

Can you cc me on the PR that disables those tests? When you re-enable the tests, remember to try them out on . :)
Let me consider if we can just skip these test for CSI, instead of reverting the entire patch.
you mean just skipping them in Alpha cluster right?
I was thinking about skipping the below test with the combination of below drivers: [Test] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode[Feature:BlockVolume] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources [Driver] csi-hostpath Won't it enough? Or, do we need to skip it only with alpha cluster?
If we just disable the tests alone and not revert the feature, we are just masking the issue. Ideally if we are not able to fix the tests and reenable them the feature should be reverted from 1.13 as well. Are you also actively working on fixing the tests and this skip is just to unblock PRs in the meantime? if so can we timebox by when we can have the tests fixed and re-enabled / the feature reverted? My fear is we might forget the fact that its disabled since there's no active reporting on it.
Actually, there is no feature in either of culprit commits. It tests or intended to improve the way it is tested. Therefore, it must be a test issue or another issue which is found by the test , not an issue introduced by the commits themselves. In the code, there is a clear comment that the test is skipped due to this issue with the reference to this issue#, so this won't be forgot. If you still worry about this issue is not tracked, we might be able to keep this issue open.
IIUC the test that were failing (and are being reverted) were for the new CSI festures going to Alpha in 1.13, right? so even these 2 culprit commits do not contain feature work, it is testing some new code additions to 1.13 and now those will be untested until we re-enable these tests. Is there an ETA to re-enable them, preferably sometime early next week? Yes +1 to keeping this issue open to track the skipped tests.
The two failing tests themselves are not for new alpha features that are in v1.13.

Instead, they are just to test csi driver works well as it works in existing in-tree driver. (And failed because they might be affected by some other alpha features or some of specific configurations in gce.) In addition, ~50 tests for csi driver, so it would be better not to remove all of the tests by reverting the PR, just for the 2 failing tests. I think that just reverting the patch won't help improving quality of k8s. As for ETA to solve the issue, I found difficulty in debuging in my local environment. It seems that the two tests fail in gce environment w/ AllAlpha=true, however, as I've already mentioned, at least csi-hostpath tests in my local environment w/ AllAlpha=true doesn't fail.(And I don't have test environment for gce.) Of course, I will help solving the issue next week. However, I will need someone's help for testing combinations of alpha features in gce environment to narrow down which flag really affects this issue. At least I would like to know the results: w/ MountPropagation=true,CSIPersistentVolume=true,BlockVolume=true,CSIBlockVolume=true all alpha flags except CSIDriverRegistry and CSINodeInfo
/reopen Thanks for the detailed explanation. I think the revert is fine in this case then. I am keeping this issue open however to track and fix the failures in these 2 tests can any of you help answer w.r.t alpha features on gce environment. thanks
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can help debug when I'm back on Tuesday. In the meantime, I would suggest looking through the failing test logs to try to triage what's going on
Here's one failing run:
I looked into the csi-hostpath failure log. As seen in the for , where the pod which has a csi-hostpath volume is being attached is scheduled, mount volume fails with these errors (1). Related code to this log is (2), so the driver doesn't seem to be registered to . As for the registration code for csiDriver, it is called as (3).

Therefore, will be logged if the driver is triggered for registration. As seen (4), registration is only triggered in , which is different from the one the pod is scheduled. In addition, driver-registrar seems to be deployed only on the node where registration is triggered (5). Is it right behaviour, while registrar is deployed as daemonset? I guess that just this fact seems to cause an issue to make csi-hostpath driver work properly. (Actually, I only tested with 1 node environment, so this might only happen in multiple nodes environment.) <details- (1) (2) Result of grep to check whether registrar is deployed to each node [ for bootstrap-e2e-minion-group-50r7][ for bootstrap-e2e-minion-group-k781][ for bootstrap-e2e-minion-group-z5cg] </details
The hostpath plugin should be deployed as a DaemonSet, so it's strange that it's only getting registered on only one node.
/assign looking into this as well
could you link the PR for skipping [external-gcepd] tests into this issue when you do that as well
skipping gcePD-external is here:
Since gce-cos-alphafeatures is now passing, do we need to keep this issue open? Should it morph into a non-blocking issue for longterm work on a better test?
We have root caused 2 of the 3 failures, which were problems in the test and test conifg. Fixes will be out soon. Still triaging the 3rd failure, which could be a real bug in alpha clusters
I've root caused csi-hostpath failure issue. It just failed to set the node properly in test code side. I will make a PR to fix it.
Or should it be done in your PR too, to avoid modifying twice in the separate PR? Actually, the fix are as below: (Failing to set node before pod creation)
Created for csi-hostpath fix, to prepare for the case that we need a separate PR. We can discuss more on this there, if needed.
/unassign found possible root cause for GCE issue: CSI NodeID annotation is not present on the Node object, could be caused by failed retries on updating Node object.

(apparently when alpha features are open we shouldn't be looking at the Node object anyway except we seem to be, so that might also be a bug). has the most context for the fix /assign
The symptom of the remaining test failure is the test pod fails to create because it fails to attach with the error: . The test succeeds the first time it's run in a cluster, and always fails after that. Two issues caused this error message. the CSI external-attacher tries to fetch the object to get the NodeID, it does so through an informer. The first time the test is run, objects are created for the very first time, and the informer returns the correct object. In subsequent test runs, s from the first run is kept, and for some reason the informer fails to detect the object and returns empty. Making the API call directly works, I'll patch the external-provisioner with this to mitigate the issue, and this is sufficient to get the test to pass. The informer bug will need to be eventually fixed. fails to post the NodeID annotation sometimes, likely due to node status patch not being retried correctly. This is crucial for clusters without feature flag set, and this is the default for 1.13. Will add the appropriate retry mechanism.
Opened to track the informer issue
Are and potential PR for the only 2 fixes being actively worked on for this issue?
Only is needed. We have a workaround for
will you have a followup PR to enable the CSI alpha tests back? or is that already in flight?
also reenables the tests
OK with merged we should see the CSI alpha tests re-enable. to keep an eye out for CI signal
The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! /close
Closing this issue. <detailsIn response to : The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The tests are actually running (and passing) in the normal suite now, now that the raw block feature moved to beta.


Question: clone, err := conversion.NewCloner().DeepCopy(&claim) if err != nil { glog.Errorf("error cloning claim %q: %v", claimToClaimKey(&claim), err) <ins> continue </ins> } claimClone := clone.(*api.PersistentVolumeClaim) ctrl.storeClaimUpdate(claimClone)

As part of the Kubernetes upgrade workflow we have in place, we are upgrading kubernetes from 1.15.3 -1.18.15 (both versions and all versions in the middle are custom patched for different behavior of the and . None of them having anything to do with kube-proxy's IPVS), and as part of this upgrade we are also migrating the from to mode. We have calico 3.12 running on the cluster as part of the CNI stack. Once the pods are migrated from to mode by applying the new DaemonSet manifest, the cluster has a series of stale rules for chain left around that breaks the runtime state of the pods. have cluster running in HA mode (3 Node cluster) where all nodes of the cluster are running as leaders and take the workload as well. There is no non-leader Pods. have customized the kubernetes and calico infra to enable the usage of Link local IP address in range for internal usecases Most of the IPs and the names have been mangled. Stale IP table rules to be cleaned up once the pods and endpoints are stable with all endpoints in Ready address mode. down the pods on a given namespace to 0 (leave the service and Endpoints around) the control plane to 1.18.15 (take the intermediate hop path. 1.15.3 -1.16.7 -1.17.5 -1.18.15) apply the kube-proxy manifest provided above. up the pods scaled down in Step 1. Current workaround we have been putting it use the as an init container and run the and then bring up the in IPVS mode and that seem to take care of the stale entries (but this seem to be too much of a big hammer approach of cleaning up) Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools: kubeadmNetwork plugin and version (if this is a network-related bug):Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label area/kube-proxy Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig sig/network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
I don't think that the live migration between kube-proxy implementations iptables<-ipvs is supported, at least I never heard about that and seems tricky. you use to have all the historical context, what do you think?
We tried implementing a cleanup mode but it was problematic and removed. (see ) You can run but that's pretty coarse. Maybe we should add a value to that flag to say which mode to try to cleanup? PRs welcome. The simplest answer is to reboot.
Let's make this a docs issues - we should write this down
can we merge the above so that we can close this issue?
not just yet :-) wouldn't signal registration and calling clean-up would at least clean things up on graceful shutdown? Granted rules will not be there until kube-proxy is restarted.


Question: "--enable-aggregator-routing=true", "--enable-logs-handler=false", "--enable-swagger-ui=true", <del> "--endpoint-reconciler-type=" + string(reconcilers.MasterCountReconcilerType), </del> <ins> "--endpoint-reconciler-type=" + string(reconcilers.LeaseEndpointReconcilerType), </ins> "--etcd-quorum-read=false", "--etcd-keyfile=/var/run/kubernetes/etcd.key", "--etcd-certfile=/var/run/kubernetes/etcdce.crt",

ref: and An e2e test should be , and the default should be updated. /assign cc
/assign
Thanks I can take this on.
Could someone point me to where the manifests are for the e2e test suites, I am happy to submit the patches?
Are there API server e2e tests currently?
there are not HA API server e2e tests that I'm aware of
I opened a PR for the default changes. Are there suggestions on a good E2E test to write if we don't have HA API server tests?
what is needed for this issue to complete? Do you think you'll get that by Monday, which is Code Freeze?
We need some E2E testing or add an integration test as defined . It will not be done today.
so are you planning to move it to 1.11, then? It's about to be automatically kicked out of 1.10 by automation.
I do not have permission to change the milestone. If you could kindly change it, that would be awesome!
[MILESTONENOTIFIER] Milestone Removed From Issue Important: This issue was missing the label for more than 7 days. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Reopening as this was reverted in


Question: newStatefulSetClone.Spec.Template = oldStatefulSet.Spec.Template               // +k8s:verify-mutation:reason=clone newStatefulSetClone.Spec.UpdateStrategy = oldStatefulSet.Spec.UpdateStrategy   // +k8s:verify-mutation:reason=clone newStatefulSetClone.Spec.MinReadySeconds = oldStatefulSet.Spec.MinReadySeconds // +k8s:verify-mutation:reason=clone <del> if utilfeature.DefaultFeatureGate.Enabled(features.StatefulSetStartOrdinal) { newStatefulSetClone.Spec.Ordinals = oldStatefulSet.Spec.Ordinals // +k8s:verify-mutation:reason=clone } </del> <ins> newStatefulSetClone.Spec.Ordinals = oldStatefulSet.Spec.Ordinals               // +k8s:verify-mutation:reason=clone </ins> newStatefulSetClone.Spec.PersistentVolumeClaimRetentionPolicy = oldStatefulSet.Spec.PersistentVolumeClaimRetentionPolicy // +k8s:verify-mutation:reason=clone if !apiequality.Semantic.DeepEqual(newStatefulSetClone.Spec, oldStatefulSet.Spec) { <del> if utilfeature.DefaultFeatureGate.Enabled(features.StatefulSetStartOrdinal) { allErrs = append(allErrs, field.Forbidden(field.NewPath("spec"), "updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden")) } else { allErrs = append(allErrs, field.Forbidden(field.NewPath("spec"), "updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden")) } </del> <ins> allErrs = append(allErrs, field.Forbidden(field.NewPath("spec"), "updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden")) </ins> } return allErrs

This applies to StatefulSetStartOrdinal feature gate. Field validation should be done regardless of the feature gate state. From
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig apps


Question: event := recorder.makeEvent(ref, annotations, eventtype, reason, message) event.Source = recorder.source <del> go func() { // NOTE: events should be a non-blocking operation defer utilruntime.HandleCrash() recorder.Action(watch.Added, event) }() </del> <ins> // NOTE: events should be a non-blocking operation, but we also need to not // put this in a goroutine, otherwise we'll race to write to a closed channel // when we go to shut down this broadcaster.  Just drop events if we get overloaded, // and log an error if that happens (we've configured the broadcaster to drop // outgoing events anyway). if sent := recorder.ActionOrDrop(watch.Added, event); !sent { klog.Errorf("unable to record event: too many queued events, dropped event %#v", event) } </ins> } func (recorder *recorderImpl) Event(object runtime.Object, eventtype, reason, message string) {

The standard event recorder, when recording an event, calls the underlying Broadcaster in a goroutine, so that even recording is non-blocking. Things like leader election emit events over the course of their lifecycle via this broadcaster. Consider the following scenario: GR1: spawn leader election ----GR2 GR2: Leader election: emit event "Lease Acquired" ----spawn GR3 () GR1: Shutdown leader election ----blocks until GR2 is done GR1: Shutdown broadcaster --() GR3: ----send on closed channel () Practically, we've observed this behavior popping up occaisionally in tests in controller-runtime:
/sig api-machinery /kind bug
<details<summaryrelevant race detector output/stack traces</summary</details
were you planning on sending a fix, or need our help to send one to fix it?
I can probably try to figure out a fix if you want, but the whole model for how event broadcasters currently work is inverse to how Go suggests you manage channels (only writers should ever close channels, not readers). I'll need to do a bit of playing around with things.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think this is fixed by . Please reopen if it's not the case.
/triage accepted
Would be possible to cherry-pick this fix in 1.20?


Question: } parentFD = childFD childFD = -1 <del> } // Everything was created. mkdirat(..., perm) above was affected by current // umask and we must apply the right permissions to the last directory // (that's the one that will be available to the container as subpath) // so user can read/write it. This is the behavior of previous code. // TODO: chmod all created directories, not just the last one. // parentFD is the last created directory. </del> <ins> // Everything was created. mkdirat(..., perm) above was affected by current // umask and we must apply the right permissions to the all created directory. // (that's the one that will be available to the container as subpath) // so user can read/write it. // parentFD is the last created directory. </ins> <del> // Translate perm (os.FileMode) to uint32 that fchmod() expects kernelPerm := uint32(perm & os.ModePerm) if perm&os.ModeSetgid > 0 { kernelPerm |= syscall.S_ISGID } if perm&os.ModeSetuid > 0 { kernelPerm |= syscall.S_ISUID } if perm&os.ModeSticky > 0 { kernelPerm |= syscall.S_ISVTX } if err = syscall.Fchmod(parentFD, kernelPerm); err != nil { return fmt.Errorf("chmod %q failed: %s", currentPath, err) </del> <ins> // Translate perm (os.FileMode) to uint32 that fchmod() expects kernelPerm := uint32(perm & os.ModePerm) if perm&os.ModeSetgid > 0 { kernelPerm |= syscall.S_ISGID } if perm&os.ModeSetuid > 0 { kernelPerm |= syscall.S_ISUID } if perm&os.ModeSticky > 0 { kernelPerm |= syscall.S_ISVTX } if err = syscall.Fchmod(parentFD, kernelPerm); err != nil { return fmt.Errorf("chmod %q failed: %s", currentPath, err) } </ins> } return nil }

What happened: Adding a VolumeMount to a container with a multi-part subpath results in only the leaf directory having 777 permissions. If you define e.g. then within the volume, and will have permissions, whereas will have permissions. This means that if you want to remove and eventually from within a pod, you will need to use a privileged pod. If the sub-directories already exist, their permissions are unchanged, and the order in which these subpaths are created appears to be the order in which they appear within the spec -- if we see first, then will be created with 755 permissions and will be created with 777 permissions, even if there's another volumeMount for further down. What you expected to happen: Sub-directories created to support a volumeMount should all have the same permissions as the leaf directory. How to reproduce it (as minimally and precisely as possible): the yaml below. The first example has volume mounts so that is created before ; the second example has volume mounts in reversed order, so is encountered before , which results in being unwritable. <details<summaryThis pod will complete</summary</details<details<summaryThis pod will fail</summary</detailsAnything else we need to know?: I believe this is the return of Environment: Reproduces on minikube and a (OpenShift) development cluster I have access to Kubernetes version (use ): minikube: OpenShift cluster: output: Cloud provider or hardware configuration: Minikube: v1.15.1; docker driver; hostpath volumes (all default) OpenShift cluster: AWS, EBS volumes OS (e.g: ): Minikube: Fedora 32 OpenShift cluster: RHEL 8.2 Kernel (e.g. ): Minikube: OpenShift cluster: Install tools: minikube v1.15.1 / Not sure Network plugin and version (if this is a network-related bug): N/A Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig storage
/assign
GitHub didn't allow me to assign the following users: mengjiao-liu. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: //NewRandomNameCustomResourceDefinition generates a CRD with random name to avoid name conflict in e2e tests func NewRandomNameCustomResourceDefinition(scope apiextensionsv1beta1.ResourceScope) *apiextensionsv1beta1.CustomResourceDefinition { <del> gName := names.SimpleNameGenerator.GenerateName("foo") </del> <ins> // ensure the singular doesn't end in an s for now gName := names.SimpleNameGenerator.GenerateName("foo") + "a" </ins> return &apiextensionsv1beta1.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: gName + "s.mygroup.example.com"}, Spec: apiextensionsv1beta1.CustomResourceDefinitionSpec{

Seeing flakes on master: https://k8s- https://k8s- and on unrelated PRs: https://k8s-
I was able to reproduce this locally (although it's very hard to trigger). Initial thoughts are that it's different from ; in this case, the actual sync is instant (millseconds) but the CRD created in the test fails to appear in discovery calls over the course of repeated attempts every 5 seconds for the remainder of the test. Normally, the type shows up in discovery almost immediately after creation.
Looks like PR might be hitting a related flaky test. I get 6 failures in 60 runs from the command I'm super new around here so thought it prudent to add the information I have found before attempting to dig deeper. I followed instructions in to build from within directory If the test failure logs are of any use to anyone just message and I'll post one of them.
Thanks. I've also been able to reproduce locally and am debugging it today. I did have to hack up the integration test setup to get apiserver logs in this context. I've also opened to explore making the integration test script more configurable with regards to logging.
I was wrong about the discovery aspect; the CRD is coming back from discovery, but the restMapper fails to map it. I also notice we're using DiscoveryInterface.ServerPreferredResources which is not actually implemented with a cache in memCacheClient.
okay, now I'm getting somewhere. I made a which exercises discovery and REST mapping without the GC in the picture at all, and have been able to reproduce the flake 1 time out of 40 runs so far with .
First problem is pretty clear in hindsight; we reset the restmapper, then get deletable resources using the discovery client directly, then sync based on the resources from discovery. The direct discovery call doesn't use the client's internal cache, and so the restmapper might be out of sync with the subsequent discovery call, causing us to sync with resources not yet represented in the restmapper's cache. That explains the isolated flake in the test I made, but doesn't yet explain why the restmapper would continue to fail on subsequent sync calls (since the nest restmapper reset would usually bring it up to date with the resources returned from the direct discovery call).

I'm out tomorrow, so a quick update here in case somebody else looks into it... can reproduce the issue where DeferredDiscoveryRESTMapper sometimes becomes unable to map the CRD resource, even after the type is available from discovery. I've also ruled out the memCacheClient by replacing it with a non-caching implementation like: And using that in the setup like: The issue can still be reproduced that way. Looks like more debugging of DeferredDiscoveryRESTMapper itself is required.
Finally tracked this down. Custom resource kinds are always given a plural suffix of "s". If the singular name ends in "s", the assumes the plural form is an "es" suffix, which will never match the generated kind. This explains the intermittent nature of the failure (the randomly generated names often happen to match the assumptions of the rest mapper).
is the root cause
Good effort


Question: // SampleAndWaterMarkObserverGenerator creates TimedObservers that // populate histograms of samples and low- and high-water-marks.  The // generator has a samplePeriod, and the histograms get an observation <del> // every samplePeriod. </del> <ins> // every samplePeriod.  The sampling windows are quantized based on // the monotonic rather than wall-clock times.  The `t0` field is // there so to provide a baseline for monotonic clock differences. </ins> type SampleAndWaterMarkObserverGenerator struct { *sampleAndWaterMarkObserverGenerator } type sampleAndWaterMarkObserverGenerator struct { clock        clock.PassiveClock <ins> t0           time.Time </ins> samplePeriod time.Duration samples      *compbasemetrics.HistogramVec waterMarks   *compbasemetrics.HistogramVec

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The apiserver is always restarting and I can't access it with any commands looks like . When I check the kubectl's status, the tells me that: And then I checked the container's status and log of the apiserver: status: log: I see that there is a line: And then I get the source code: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.19.3. But I can't use the now because of the problem, so I can't give you a formal result about the version. Cloud provider or hardware configuration: Ryzen 5 4500U 16GB memory 30GB SSD In the VMware Workstation 15.5.6 OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
The complete version for the log of apiserver:
/cc
: did you tweak that machine's clock backwards, or run any software that does this?
Time calculations should be written to use go's , not count on wall clock time always moving foward.
: The proximal problem is that uses and gets the answer that time went backwards.
sure, but should be immune to wall clock resets per
Exactly. It looks to me like this test for ordering is using the monotonic clock and somehow that is saying time went backwards.
does the quantization () factor into well-ordered detection? I'm not sure how the monotonic clock interacts with Unix / UnixNano
No, I see the quantization is buggy, it uses wall clock. But the ordering test does not depend on the quantization.
/triage accepted /priority urgent-critical
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /triage accepted /priority urgent-critical </details
very odd that this is repeatable. /priority critical-urgent
: it is very odd that this is repeatable. Do you have anything unusual in your environment ?

I see it is vmware workstation. Does that mess with the clock in any unusual way? I also wonder about the golang runtime; did you do your own build (if so, what version of golang) or fetch binaries from somewhere (where?); full output would be helpful.
I don't have anything unusual in my environment. I run it in the ubuntu driven by vmware workstation, because my real machine's operation system is Windows10 , and I need a virtual machine to run k8s. My computer is a laptop computer. I got the golang runtime by apt-get. I also got the distribution of k8s by apt-get too. I did not do my own build. ! Okay, I got my golang and k8s version now: !
I have exactly same problem in my local lab. Virt infra on Virtual Box 6.1.16 Guest OS
you try and, if so, does it fix your problem and does it log complaints about the monotic clock going backwards? you tell us about your host and the settings on your VM(s)?
exactly matches same error as posted by author of issue, I have NOT tried VM settings: 4 vCPUs 4GB RAM swap is OFF on OS level. Host HW: Core i9 9900K 32GB Ram BTW: I have managed to install 1.18.12 on same VM eventually - cause for me it is just lab infra to study, so 1.19.X is NOT that important for me to have on this infra.
: host OS? Is your VB "Version 6.1.16 r140961 (Qt5.6.3)"? VM settings match the following? <img width="616" alt="Screen Shot 2020-11-16 at 4 56 10 PM" src="https://user-"<img width="742" alt="Screen Shot 2020-11-16 at 4 55 58 PM" src="https://user-"<img width="720" alt="Screen Shot 2020-11-16 at 4 55 45 PM" src="https://user-"
OS Name: Microsoft Windows 10 Enterprise OS Version: 10.0.

N/A Build OS Manufacturer: Microsoft Corporation OS Configuration: Standalone Workstation OS Build Type: Multiprocessor Free ! ! ! !
I tried to reproduce the problem using Ubuntu 18.0.4.5 and Kubernetes 1.19.4 (installed by kubeadm, using Calico as the network plugin), but the problem did not strike. The guest was created from the Ubuntu live Server ISO with a very plain install, picking only OpenSSH from the things to add during the install. No explicit tweaks regarding timekeeping. In the guest: I am also running the exact same version of VBox as reported above. I have the ExtensionPack of the same version installed. My host is Windows 10 Pro version 2004. Intel Core i9-9900 (8 hyperthreaded cores, thus 16 logical processors), 32 GB RAM. Configured for automatic time sync. Last sync 34 minutes ago, with server
Same issue in my env: VM (CentOS 7.9.2009 4 CPU+12G Memory) on virtualbox(6.1) Host is Windows 10 (16G Memory, CPU Intel(R) Core(TM) i3- CPU @ 3.60GHz 3.60 GHz) docker version: docker-ce-19.03.9-3.el7.x8664 K8S version: 1.19.4 Part error log output from docker logs -f <kube-apiserver-containerid:
Hi, same here vitualbox 6.1.18 latest Linux master-1 5.4.0-72-generic latest /etc/os-release NAME="Ubuntu" VERSION="20.04.2 LTS (Focal Fossa)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 20.04.2 LTS" VERSIONID="20.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" PRIVACYPOLICYURL="" VERSIONCODENAME=focal UBUNTU_CODENAME=focal tried with K8s 1st with 1.21.0-00 then 1.19.10-00 same Time went backwards message in the logs. Api server uses large amount of CPU at doing nothing. etcd cluster seems also to be totally lost. 3 masters setup.

thanks
Hm ... this must have something to do with the fact we're running in a VM. Funny enough ... you guys are reporting apiserver crashing. What led me to this issue is that my metrics-server crashes with this same log message instead :). I originally had ntpd installed on the VM, but removing it did not help. So ... if time goes backward, it must be something that VirtualBox does. I'll try to see if I can repro with HyperV. EDIT: I was able to get rid of this error with VirtualBox by using to disable some kind of automated clock sync that VirtualBox does by default. Don't forget to use NTPd, as you will need some time synchronization :). EDIT 2: Spoke too soon. That did not really help. It seems to have slightly delayed the issue, though, as it only presented itself after a couple of minutes... EDIT 3: So, the reason why it's still syncing time, is VBoxGuestAdditions. When they are running, the guest synces time with the host, even with the above time-sync setting disabled. One either has to stop the guest additions or pass them (but ... it's not really a configuration change: ). EDIT 4: The above doesn't seem to have an effect at all. Still, according to the metrics server, time goes backwards. Well ... let's try to completely disable the VirtualBox guest additions: (for my use case, this is totally fine, as I don't need any of the guest additions features such as shared folders or time sync). EDIT 5: Nothing helps. I reckon, either the check for "time went backwards" is somewhat buggy, or some odd time sync still takes place. For now, because I don't really need it, I am just disabling the metrics server. EDIT 6: Huh. Got it, finally. Never surrender, I guess :). Turns out, you gotta attack it from all sides. (see above) Add the argument to (see above) (this is new; turns out, even if one tells guest additions not to sync time, and disables the time syncing on the VM, something can still sync the time -- this setting makes the guest OS completely blind to the fact it is running in a VM).

I suggest to keep ntp and guest additions (with the disabled time sync) on at all times. EDIT 7: Also ... do no let your host sleep. When it sleeps and then wakes up, a lot of k8s cannot cope with it and begins rolling. It should recover, but if you can avoid sleeping, do. For me, this actually meant installing and using "Don't sleep" (alternative to "Caffeine" or similar) to keep it awake. For some reason, when you lock your computer, Windows will try to sleep. It should be possible to disable this mechanic (), but even though I did, it would still sleep. That's why I ended up with "Don't sleep". With all of this in place, it finally works as expected. None of the k8s' pods roll anymore.
I had exactly the same issue. Followed the steps listed above and it solved my issue. Thanks , much appreciated!


Question: # Karma output www/test_out <ins> # precommit temporary directories created by ./hack/verify-gendocs.sh and ./hack/lib/util.sh _tmp/ doc_tmp/ </ins>

When I passing to kubelet, it did not respect the option, and still copy the to the pod. What makes it special is that the provided by myself is empty. After skimming the code, I found the logic here is problematic: Because is empty, an empty array and no error is returned. Then, if the dnsPolicy is (that is the default config for CoreDNS deployment), the turns to be : Then, containerd will get an empty , so it will copy from the host to the container: Finally, CoreDNS reports forwarding loop error. Kubelet should refuse to use the wrong empty to create a pod, and complains with errors. to kubelet, and make sure the specified file is empty. the output of CoreDNS, and its . No response_ <detailsk8s v1.28.2 containerd v1.7.6 </details<detailsNone </details<detailsAny major Linux distribution. </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/sig network
why if someone wants to pass an empty to the pods? is that a possibility? I do not know if kubelet should compensate what it seems an operator problem, or am I reading it wrong?
This situation MAY occur when systemd-resolved is used, and the operator pass the to kubelet. Although systemd-resolved create this file in manner of "create temp, write temp, rename" ... ... the UPLINK config () file may still be full of comments if there is no available name server at the time.
And after a while, systemd-resolved got an available dns server by any way, then the UPLINK config should be recreated, however the kubelet have taken the stale and full-of-comments .

I see there is a race there, thanks for the explanation ... the only question I have is if someone is relying on this behavior of passing an empty to the pods, a change like this will break it
/remove-sig node. This is on network team to decide how to migrate a new behavior for empty file..
Those labels are not set on the issue: <detailsIn response to : /remove-sig node. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig node
If set , it means will use the default value , so this change may be not a break, but should be a warning.
It is suggesting to change the behavior to stop the pod creation
/assign But if already have design on this, please go ahead.
I still don't have clear what is the proposal and how to maintain compatibility
Cool, may be it's like new warning events just like , or something else.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen
This is a fun bug, but not that hard to comprehend, I think. Forget whether it's a "good idea" or not. Is it valid to pass an empty file to ? If yes, should it mean the same as not passing that flag or should it mean "no config". I think it IS valid and it SHOULD mean "no config". What really happens is that kubelet gets empty slices (but no error, so not ambiguous). When it creates the DNSConfig message, it appears empty, because it is comprised entirely of fields with no instances. That is ambiguous - it could mean "no data" or "not specified". It's unclear to me if we can tell the difference between those? - do you know? can return nil.

Containerd further conflates those two by parsing into a string, and assuming means "not specified". Even if it WAS specified! Then, containerd "helpfully" decides that, if you don't specify any contents, it should copy the host's I am not 100% confident in CRI semantics - is it valid for CRI to NOT populate the message? Or is that effectively required? If a CRI caller does not specify , is the CORRECT behavior to go look at the host? That seems like over-stepping, to me. If I wanted a DNS config, I would have sent it. Agree or disagree? So IMO, the fix here is either (in my pref order): 1) containerd should not fall back on host's AT ALL 2) containerd should differentiate "not provided" from "provided and empty" ( returned nil) 3) we should add a "this struct intentionally left blank" marker to , so containerd can do the above So I think this very much is a SIG Node issue. /sig node As to the question of how this problem was are you setting the kubelet to an empty rather than just configuring the specific Pods in question?
This is because CoreDNS reports a forwarding loop issue due to systemd-resolved is used in the node, and I pass to kubelet. But it has a race condition here when the node restarted, the file may be full of comments if there is no available name server at the time. And after a while, systemd-resolved got an available dns server by any way, then the UPLINK config should be recreated, however the kubelet have taken the stale and full-of-comments . Of course passing an empty is incorrect, but it is not deliberately.
Fixed in containerd
you should check if you need to fix this too in cri-o ^^^
It looks like . I'm going to create a PR in .
Thank you and thanks for the hint! :heart:


Question: // serverPreferredResources returns the supported resources with the version preferred by the // server. If namespaced is true, only namespaced resources will be returned. func (d *DiscoveryClient) serverPreferredResources(namespaced bool) ([]unversioned.GroupVersionResource, error) { <del> results := []unversioned.GroupVersionResource{} serverGroupList, err := d.ServerGroups() if err != nil { return results, err } </del> <ins> // retry in case the groups supported by the server change after ServerGroup() returns. const maxRetries = 2 </ins> var failedGroups map[unversioned.GroupVersion]error <del> for _, apiGroup := range serverGroupList.Groups { preferredVersion := apiGroup.PreferredVersion groupVersion := unversioned.GroupVersion{Group: apiGroup.Name, Version: preferredVersion.Version} apiResourceList, err := d.ServerResourcesForGroupVersion(preferredVersion.GroupVersion) </del> <ins> var results []unversioned.GroupVersionResource RetrieveGroups: for i := 0; i < maxRetries; i++ { results = []unversioned.GroupVersionResource{} failedGroups = make(map[unversioned.GroupVersion]error) serverGroupList, err := d.ServerGroups() </ins> if err != nil { <del> if failedGroups == nil { failedGroups = make(map[unversioned.GroupVersion]error) } failedGroups[groupVersion] = err continue </del> <ins> return results, err </ins> } <del> for _, apiResource := range apiResourceList.APIResources { // ignore the root scoped resources if "namespaced" is true. if namespaced && !apiResource.Namespaced { </del> <ins> for _, apiGroup := range serverGroupList.Groups { preferredVersion := apiGroup.PreferredVersion groupVersion := unversioned.GroupVersion{Group: apiGroup.Name, Version: preferredVersion.Version} apiResourceList, err := d.ServerResourcesForGroupVersion(preferredVersion.GroupVersion) if err != nil { if i < maxRetries-1 { continue RetrieveGroups } failedGroups[groupVersion] = err </ins> continue } <del> if strings.Contains(apiResource.Name, "/") { continue </del> <ins> for _, apiResource := range apiResourceList.APIResources { // ignore the root scoped resources if "namespaced" is true. if namespaced && !apiResource.Namespaced { continue } if strings.Contains(apiResource.Name, "/") { continue } results = append(results, groupVersion.WithResource(apiResource.Name)) </ins> } <del> results = append(results, groupVersion.WithResource(apiResource.Name)) </del> <ins> } if len(failedGroups) == 0 { return results, nil </ins> } } <del> if len(failedGroups) > 0 { return results, &ErrGroupDiscoveryFailed{Groups: failedGroups} } return results, nil </del> <ins> return results, &ErrGroupDiscoveryFailed{Groups: failedGroups} </ins> } // ServerPreferredResources returns the supported resources with the version preferred by the

Split off from , the thing that actually caused the flakes throughout the system (as opposed to just failing the TPR test) is (https://k8s-) Which I guess means that the discovery client got a list, but then the TPR was gone when it tried to get details? This is not an emergency since we've marked the test flaky, but we should fix it.
The print out of the error message included the message, which was generic 404. So I believe returned a 404 without a JSON body.
Yes. The discovery client returns a partial result when this error occurs, see . I think this behavior is correct. Caller should look into the error and ignore it or retry the discovery.


Question: // Make an event that kubelet restarted. // TODO: get the real minion object of ourself, // and use the real minion name and UID. <ins> // TODO: what is namespace for node? </ins> ref := &api.ObjectReference{ <del> Kind:      "Minion", </del> <ins> Kind:      "Node", </ins> Name:      kl.hostname, UID:       types.UID(kl.hostname), <del> Namespace: api.NamespaceDefault, </del> <ins> Namespace: "", </ins> } kl.recorder.Eventf(ref, "starting", "Starting kubelet.") }

now shows NodePhase, NodeCondition, etc. information from NodeStatus. It should also show related events like what we did for pods.


Question: // Schedule tries to schedule the given pod to one of node in the node list. // If it succeeds, it will return the name of the node. <del> // If it fails, it will return a Fiterror error with reasons. </del> <ins> // If it fails, it will return a FitError error with reasons. </ins> func (g *genericScheduler) Schedule(pod *v1.Pod, nodeLister algorithm.NodeLister) (string, error) { trace := utiltrace.New(fmt.Sprintf("Scheduling %s/%s", pod.Namespace, pod.Name)) defer trace.LogIfLong(100 * time.Millisecond)

Fix this TODO (raised by : Part of: One thoughts to fix this is, we can redesign the equivalence cache data structure to: bool instead of deleting the entry. certain version tag in the entry, update the version per WRITE op (Update and Invalidate). the version after eCache predicate, and check it before update eCache.
Another possible approach is: eCache before running a predicate. the predicate. the eCache for the predicate. eCache. Basically, in the existing code all predicates are run first and their results are collected and then at the end of the function the eCache is updated. Instead of that, we can move the eCache update for each predicate inside the for loop and next to each predicate execution and synchronize before start of the execution of each predicate. The downside to my suggested approach is that performance may suffer due to frequent locking and unlocking.
Prefer to bsalamat@ 's option which is simpler :). Comparing to performance improvement by eCache, the lock is acceptable to me.
yup same here, going with the locking is easier and is acceptable. - also using a flag for invalidation might just move the problem from eCache to the flag ?
Oh I missed this notification somehow, busy on cleaning scheduler integration test. Yep so if locking is acceptable, it should be cleanest way to fix this, optimistic control would be a more complicated story.
- are you handling this one ?
Already been taken by volunteer days ago :)


Question: "path/filepath" "strconv" "strings" <ins> "time" </ins> "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource"

Node Allocatable setup test has been flaking recently. Example flake: https://k8s- /sig node cc
The kubelet logs look fine. After speaking with we are querying the node api object immediately after the kubelet configuration is updated. At this point, the kubelet may not have updated its node status, meaning we see the old node status rather than the new, correct one.


Question: cronJobControl: tt.fields.cronJobControl, } jm.now = justASecondBeforeTheHour <del> now := time.Now() then := time.Now() wg := sync.WaitGroup{} wg.Add(1) go func() { now = time.Now() jm.queue.Get() then = time.Now() wg.Done() return }() </del> jm.updateCronJob(&cj, &newCj) <del> wg.Wait() d := then.Sub(now) if d.Round(tt.roundOffTimeDuration).Seconds() != tt.deltaTimeForQueue.Round(tt.roundOffTimeDuration).Seconds() { t.Errorf("Expected %#v got %#v", tt.deltaTimeForQueue.Round(tt.roundOffTimeDuration).String(), d.Round(tt.roundOffTimeDuration).String()) </del> <ins> if tt.fields.queue.t.Seconds() != tt.deltaTimeForQueue.Seconds() { t.Errorf("Expected %#v got %#v", tt.deltaTimeForQueue.Seconds(), tt.fields.queue.t.Seconds()) </ins> } }) }

pull-kubernetes-unit TestController2updateCronJob/spec.schedulechanged TestController2_updateCronJob /sig apps /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is important to resolve before promotion to v1
looks like the cause was
This function should be related.
Another one:
When the flake happen, there are messages like When the system is overloaded, it would happen. is trying to replace the current queue in tests with a mock.
Yeah, I spoke with to make sure we use a mock of the queue so we only get the most important bit of information ignoring the remaining parts


Question: return context.Background() } <ins> // TestConcurrentlyCreateAndDeleteContainers is a regression test for #93771, which ensures // kubelet would not panic on concurrent writes to `dockerService.containerCleanupInfos`. func TestConcurrentlyCreateAndDeleteContainers(t *testing.T) { ds, _, _ := newTestDockerService() podName, namespace := "foo", "bar" containerName, image := "sidecar", "logger" const count = 20 configs := make([]*runtimeapi.ContainerConfig, 0, count) sConfigs := make([]*runtimeapi.PodSandboxConfig, 0, count) for i := 0; i < count; i++ { s := makeSandboxConfig(fmt.Sprintf("%s%d", podName, i), fmt.Sprintf("%s%d", namespace, i), fmt.Sprintf("%d", i), 0) labels := map[string]string{"concurrent-test": fmt.Sprintf("label%d", i)} c := makeContainerConfig(s, fmt.Sprintf("%s%d", containerName, i), fmt.Sprintf("%s:v%d", image, i), uint32(i), labels, nil) sConfigs = append(sConfigs, s) configs = append(configs, c) } containerIDs := make(chan string, len(configs)) // make channel non-blocking to simulate concurrent containers creation var ( creationWg sync.WaitGroup deletionWg sync.WaitGroup ) creationWg.Add(len(configs)) go func() { creationWg.Wait() close(containerIDs) }() for i := range configs { go func(i int) { defer creationWg.Done() // We don't care about the sandbox id; pass a bogus one. sandboxID := fmt.Sprintf("sandboxid%d", i) req := &runtimeapi.CreateContainerRequest{PodSandboxId: sandboxID, Config: configs[i], SandboxConfig: sConfigs[i]} createResp, err := ds.CreateContainer(getTestCTX(), req) if err != nil { t.Errorf("CreateContainer: %v", err) return } containerIDs <- createResp.ContainerId }(i) } for containerID := range containerIDs { deletionWg.Add(1) go func(id string) { defer deletionWg.Done() _, err := ds.RemoveContainer(getTestCTX(), &runtimeapi.RemoveContainerRequest{ContainerId: id}) if err != nil { t.Errorf("RemoveContainer: %v", err) } }(containerID) } deletionWg.Wait() } </ins> // TestListContainers creates several containers and then list them to check // whether the correct metadatas, states, and labels are returned. func TestListContainers(t *testing.T) {

What happened: Windows kubelet panic occasionally on nodes which have high volume of pods run and delete. Full logs: Preliminary investigate result: How to reproduce it (as minimally and precisely as possible): Run high volume of pods and delete them when they completed in Windows nodes. Environment: Kubernetes version (use ): 1.17.7 Cloud provider or hardware configuration: Azure AKS OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..1282
/assign I would like to give this a shot.
/remove-sig windows I think this is not a windows specific problem. /sig node /area kubelet
Hi! Thanks for your report, I have filed to address this issue.
This issue is a bit urgent for our situation, it occurs every few days in production clusters. Seems the PR is pending a few days, is it possible to increase the priority to merge it trunk and 1.17/1.18 branch? Thanks
I understand how urgent this issue is, but I am not privileged to do anything further :(
Do you know anyone who can help?
looks like a simple fix, add a lock for , I could help.
There is already a fix by , the problem is the PR is pending, not sure if you or someone can increase the priority to merge it trunk and 1.17/1.18 branch asap. Our AKS clusters are affecting by this frequently.
Thanks guys.