Question: SkipIfProviderIs("gke") By("Creating a job") // 50% chance of container success, local restarts. <ins> // Can't use the failOnce approach because that relies // on an emptyDir, which is not preserved across new pods. // Worst case analysis: 15 failures, each taking 1 minute to // run due to some slowness, 1 in 2^15 chance of happening, // causing test flake.  Should be very rare. </ins> job := newTestJob("randomlySucceedOrFail", "rand-non-local", api.RestartPolicyNever, parallelism, completions) job, err := createJob(f.Client, f.Namespace.Name, job) Expect(err).NotTo(HaveOccurred())

Job should run a job to completion when tasks sometimes fail and are locally restarted Expected error: <*errors.errorString | 0xc20807c990: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred
Couple of failures: from
Another failure:
Another failure:
This test was introduced by and in and . It's currently failing at a rate of about 7%. When it passes, it typically does so in 30-90 seconds. Occasionally it times out after 15 minutes. I think that there's a real bug in the system here. The timeout occurs while waiting for the Job to terminate, which it never does.
I'm going to label it as flaky until the bug has been fixed.
Another portion of flakes:
/sub
any progress on this? Is it being actively worked on?
I've been working on this today. Saw 1 e2e failure in 50 runs. Logs suggest the problem is that half of kubelet's brain thinks it started a pod, and the other half of it thinks it didn't. In fact it didn't. So the test times out waiting for the not-started pod to complete.
Update: the kubelet did start all the pods, and they all terminated, and then the job controller somehow didn't notice that they were done until 40+ minutes later.
This is interesting. There were 3 apiserver 500's around the time of the test failure, and no 500's any other time in the 8 hours the cluster was up, and many other test runs occurred. 500s:
The code that is failing is this in at line 271 the gets an error, and then decides the error is a 500.
can you say anything about the above mentioned apiserver internal error which occurs immediately on receiving a watch request?
Look in the relevant kubelet logs to see how they interpreted the error. It's a bit silly that we printed a stack trace but not the error text :( , Eric Tune wrote:
E1022 04:13:24. 3622 ] Failed to watch *api.Pod: Internal error occurred: too old resource version: 171 ()
This is happening at line 269 of Is this an unexpected condition? If not, why is it a 500?
Oh, I thought that was fixed.

It shouldn't be an internal error, you're out of the watch window and need to re-list. , Eric Tune wrote:
OK I reopened . did the kubelets involved re-list soon after getting these 500s? If so, then this is coincidence and not your problem. , Daniel Smith wrote:
In all 3 cases, they relist immediately, but then they don't re-watch for about an hour. That seems wrong, no? Relist at 3:56 and rewatch at 4:53 Relist at 4:07 and rewatch at 5:00 relist at 4:13 and rewatch at 4:55
The time reported is the end of the watch, not the beginning. The last one it seems you actually didn't get the log line for the watch, actually. But the first two look WAI. , Eric Tune wrote:
Oh, duh.
Working on which I suspect may fix this.
Okay, looking at a new test failure I see that there are expected failures, and the test just times out due to backoff by the kubelet. The following things happen: runs and fails (1) runs and fails (2) backs off for 10s runs and fails (3rd failure) backoff 20s runs and fails (4th failure) backoff 40s runs and fails (5th failure) backoff 1m20s runs and fails (6th failure) backoff 2m40s runs and fails (6th failure) kubelet backoff 5m runs and fails (6th failure) backoff 5m timeout now that at least 15m10s has elapsed in backoffs. Failures have a probability of 50%. Chance of getting 8 fails in a row is 1 in 256. Four pods per test run is 4 in 256 or 1 in 64. I see failures about 1 in 50, so that seems about right. So, I need to rewrite the test so that it writes to the emptydir after the first failure, and succeeds after that.
Ugh. My fix for local breaks non-local restarts.
Happened again.


Question: ### Clean up ### if [[ "${E2E_DOWN,,}" == "true" ]]; then <ins> # Sleep before deleting the cluster to give the controller manager time to # delete any cloudprovider resources still around from the last test. # 60 seconds was chosen as an arbitrary bound for how long it should take # to delete load balancer resources. sleep 60 </ins> go run ./hack/e2e.go ${E2E_OPT} -v --down fi

All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.

The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.

I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.


Question: return nil } <del> func (s *Server) hostnameHealthCheck(req *http.Request) error { masterHostname, _, err := net.SplitHostPort(req.Host) if err != nil { if !strings.Contains(req.Host, ":") { masterHostname = req.Host } else { return fmt.Errorf("Could not parse hostname from http request: %v", err) } } // Check that the hostname known by the master matches the hostname // the kubelet knows hostname := s.host.GetHostname() if masterHostname != hostname && masterHostname != "127.0.0.1" && masterHostname != "localhost" { return fmt.Errorf("Kubelet hostname "%v" does not match the hostname expected by the master "%v"", hostname, masterHostname) } return nil } </del> // Checks if kubelet's sync loop  that updates containers is working. func (s *Server) syncLoopHealthCheck(req *http.Request) error { duration := s.host.ResyncInterval() * 2

When looking at Kubelet healthz through proxy on master I'm getting: cc
cc
kubelet expects to be the hostname, while master uses IP in the request instead. /cc people who knows more about proxy:
Master's expectation looks wrong.
The idea of that health check is to ensure that the master schedules work to the name expected by the kubelet. Otherwise sometimes you end up in a situation where the master is scheduling work for IP and the kubelet is looking for work for 'hostname -f'; the node status says 'Ready' but no pods get started on that node. User's must use --hostname_override to correct the situation. It has been a while since I have looked at that code so things may have changed, but in this case is the master scheduling using the hostname "10.240.170.229" ? If so then pods will not start on that node. If not then the health check needs a better way of getting the hostname as the master knows it.
I don't think this is related to the proxy, I think your master has an incorrect node configuration, as says.
- I checked this on and clusters as well, with the same result. All of us were running default GCE configuration.
That's very weird, I reproduce, too. But it has nothing to do with the proxy, rather it's the requested host (????!?): So kubelet gives different answers depending on whether you refer to it as "localhost" or by its IP address.
If kubelet is doing the healthz check about the host in the HTTP request, then that's wrong. To be useful, it should be making an assertion about the Node object that represents it in the system.
To give more context, the change was introduced in I think was describing a situation where kubelet and master have discrepancy on what the node is. E.g., kubelet thinks it's node HOSTNAME, while master thinks the node is IP. In this case, kubelet wouldn't know which Node object to check. if I understand correctly, this should not happen now because kubelet would not update the node status of the incorrect node, so the node would eventually become unavailable kubelet can register itself now
Yeah the original patch in sent the host name presumed by master along with the request but we though the hostname from the header sufficed as we could not come up with a scenario where the two differ.

That can be changed back if needed. It looks like the localhost bypass was later in in your cluster if you disable the health check do pods actually start on the node ?
hmm... do you mean the master would not be able to reach the node if it has the incorrect name ? The hostname is correct it is just different from what the node uses. Say for example the host uses "" and the node node knows itself as "93.184.216.34". The health check will pass because the node is reachable, but pods don't start because work no work is scheduled for "93.184.216.34" Yeah I think that is a safe path.
kubelet updates node status now, and if the node status isn't updated for sometime, node controller would deem the node unhealthy. E.g., if the node knows itself as "93.184.216.34", it'd try to update the status of node "93.184.216.34". The status of node "" would not be updated by anyone, and would eventually become unhealthy.
It fails through the proxy because master's proxy code uses IP address (not hostname) to reach the node. Master is actually doing the right thing there. Master doesn't directly health check kubelets any more (monit does) so that's why this isn't currently breaking anything. I'm guessing the localhost exception was because that's how monit health checks; but that's evidence that the original assumption that one could use the HTTP request's host for this was faulty, and the check should be completely removed or done some other way.
got it! I see. The check is no longer correct then and given what explained the check is unnecessary :) I'll open a PR to remove it.
thanks!


Question: "//pkg/kubelet/types:go_default_library", "//pkg/security/apparmor:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", <ins> "//staging/src/k8s.io/api/scheduling/v1beta1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/api/equality:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library",

https://k8s- The following tests are broken: NVIDIA GPU Device Plugin PriorityLocalStorageEvictionOrdering PriorityMemoryEvictionOrdering Looks like some new validation was to prevent two things: PodPriority directly system-critical priority classes outside of the kube-system namespace. To fix this, we need to create new priority classes for these tests. cc /kind failing-test /sig node /sig scheduling /priority important-soon
/kind bug
/assign
This doesn't appear to have fixed the tests. Tests now panic during the Aftereach in which the PriorityClass is deleted. /reopen
Panic:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: $ kubectl create -f /tmp/secret.json secret "nginxsecret" created <del> $ kubectl create configmap nginxconfigmap --from-file=examples/https-nginx/default.conf </del> <ins> $ kubectl create configmap nginxconfigmap --from-file=examples/https-nginx/default.conf </ins> configmap "nginxconfigmap" created ```

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT on sample file I was using the /examples/https-nginx sample application for some small testing purposes and was surprised by the relative high load it kept having in the nginx pods. After some inspection I found out that kubernetes/examples/https-nginx/auto-reload- is basically running a busy loop. the command doesn't filter out some of the events it should filter out. At least on my system with just this sample config I always get a OPEN on basically immediately so I would recommend adding as arguments to the inotifywait command
I can take this.
I already have a PR in flight for this.
OK thanks for info


Question: } sharedInformers.Start(nil) <del> controller.Run(1, nil) // runs forever return nil </del> <ins> controller.Run(5, nil) // runs forever panic("unreachable") </ins> }

this is preventing nodes from starting in large and enormous clusters and is fixed by the two linked PRs.
Has this issue been resolved by the merged PRs?


Question: if len(address.Hostname) > 0 { allErrs = append(allErrs, ValidateDNS1123Label(address.Hostname, fldPath.Child("hostname"))...) } <del> // During endpoint update, validate NodeName is DNS1123 compliant and transition rules allow the update </del> <ins> // During endpoint update, verify that NodeName is a DNS subdomain and transition rules allow the update </ins> if address.NodeName != nil { <del> allErrs = append(allErrs, ValidateDNS1123Label(*address.NodeName, fldPath.Child("nodeName"))...) </del> <ins> for _, msg := range ValidateNodeName(*address.NodeName, false) { allErrs = append(allErrs, field.Invalid(fldPath.Child("nodeName"), *address.NodeName, msg)) } </ins> } allErrs = append(allErrs, validateEpAddrNodeNameTransition(address, ipToNodeName, fldPath.Child("nodeName"))...) if len(allErrs) > 0 {

introduced a new requirement for node names to be DNS1123Labels. In case of hack/local-up-, an IP address is used as node name and thus endpoints aren't created for services, e.g. kube-dns doesn't work anymore. There are other projects that use IP addresses as node names for running simplified k8s dev environment: Probably there are other projects where it's convenient to have just an IP address as a node name. Verified against . On affected installs, service latency e2e test hangs, see e.g. Example ( some newlines):


Question: } } // TODO: Once we get rid of /healthz consider changing this to post-start-hook. <del> err := s.addReadyzChecks(healthz.NewInformerSyncHealthz(c.SharedInformerFactory)) </del> <ins> err := s.AddReadyzChecks(healthz.NewInformerSyncHealthz(c.SharedInformerFactory)) </ins> if err != nil { return nil, err }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When using GenericAPIServer you can only add new checks via function which adds them to all , , and endpoints. All other methods like are private. I would like to request those methods to be made public: Why is this needed: Developers of apiservers should be able to define their readiness probes add them to specific endpoint of their choosing. For example in case of Metrics Server we defined probes: Readiness - Metrics cache is full. As MS serves data only from cache, traffic will be only sent after cache is full. Liveness - Metric scraping loop is not deadlocked. It doesn't make sense to have those probes to both endpoints. Not ready metric cache should not cause MS being killed via liveness probe. Delayed scrape should not cause traffic not being served from cache due failing readiness probe. /assign
/sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig apimachinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/cc /triage accepted
I can see this. PR welcome.


Question: BeforeEach(func() { nodes, err := framework.Client.Nodes().List(labels.Everything(), fields.Everything()) expectNoError(err) <del> numNodes = len(nodes.Items) </del> nodeNames = sets.NewString() for _, node := range nodes.Items { nodeNames.Insert(node.Name)

For example: I'm disabling the test in But we need to re-enable it, or roll-back the test.
fyi
Shouldn't this is part slow test, and the fix should be either: 1) Increase Kubernetes-e2-gke-ci timeout value if decide to include slow to that project or 2) disable all slow tests on that project? cc/ Which one is required here?
The test runs 50 pods per node, which is over the default --max-pod limit of kubelet. The test relies on the flag --max-pods=100 being set in cluster/gce/config- I will disable it for all GKE tests in On the other hand, it's not clear to me where the timeout is set for a specific build. /cc
Timeouts for each Jenkins build are in the Jenkins project configurations themselves, which are not currently saved in version control anywhere. :
Discussed with about the test offline. There are several issues: Do we want to run any kind of kubelet resource monitoring test on GKE? Is it useful? We only override max-pods in GCE. Should the test itself figure out what platform it can run based on the #pods in the test (see ) Per-test configuration allows greater flexibility (e.g., we may run 100 pods on GCE to test kubelet's scalability), but it'd make the tests more complicated as they need to be aware of the test environment (which many e2e tests already do); there is also the burden of updating the tests once the max-pod value changes. Does this test belong to SLOW tests or should it belong to a new category? Not all slow tests have the same attributes. Arguably this test could be enabled only on selective builds. The downside would be more flags -- less readable/manageable e2e configs. Not all builds which run SLOW tests have the proper timeout value to run this test. We need to manually change them every time a slow test is . Re-assigning this back to since she has more concerns about the PR.
Had offline discussion with We are going to split current Kubelet resource usage tracking test into two, one is targeted for GCE, another one is for GKE. In this case, we won't mess up the test code itself by introducing knobs. Re-assign this back to


Question: Versions:         expAPIVersions, PreferredVersion: unversioned.GroupVersion{GroupVersion: storageVersion, Version: apiutil.GetVersion(storageVersion)}, } <del> apiserver.AddGroupWebService(m.handlerContainer, c.APIGroupPrefix+"/"+latest.GroupOrDie("extensions").Group+"/", group) </del> <ins> apiserver.AddGroupWebService(m.handlerContainer, c.APIGroupPrefix+"/"+latest.GroupOrDie("extensions").Group, group) </ins> allGroups = append(allGroups, group) apiserver.InstallServiceErrorHandler(m.handlerContainer, m.newRequestInfoResolver(), []string{expVersion.Version}) }

Loading returns Looking at the code is registered in the same way as which works fine. So not sure why is this broken.


Question: import ( "errors" "os" <ins> "github.com/golang/glog" </ins> ) type Mounter struct { mounterPath string } <ins> var unsupportedErr = errors.New("util/mount on this platform is not supported") </ins> // New returns a mount.Interface for the current system. // It provides options to override the default mounter behavior. // mounterPath allows using an alternative to `/bin/mount` for mounting.

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Calls to on unsupported platforms do not report any error and potentially misbehave. What you expected to happen: I would expect an error to be reported, that can be consumed by the caller. How to reproduce it (as minimally and precisely as possible): for and were failing on darwin, and no error was being reported. See: & Environment: OS (e.g. from /etc/os-release): I have been testing on osx/darwin, which is not a supported platform.
/sig storage
/assign


Question: epSlice1DifferentRV.ResourceVersion = "rv2" testCases := map[string]struct { <del> updateParam *discovery.EndpointSlice checksParam *discovery.EndpointSlice expectHas   bool expectStale bool </del> <ins> updateParam                     *discovery.EndpointSlice checksParam                     *discovery.EndpointSlice expectHas                       bool expectStale                     bool expectResourceVersionsByService map[types.NamespacedName]endpointSliceResourceVersions </ins> }{ "same slice": { updateParam: epSlice1, checksParam: epSlice1, expectHas:   true, expectStale: false, <ins> expectResourceVersionsByService: map[types.NamespacedName]endpointSliceResourceVersions{ {Namespace: epSlice1.Namespace, Name: "svc1"}: { epSlice1.Name: epSlice1.ResourceVersion, }, }, </ins> }, "different namespace": { updateParam: epSlice1, checksParam: epSlice1DifferentNS, expectHas:   false, expectStale: true, <ins> expectResourceVersionsByService: map[types.NamespacedName]endpointSliceResourceVersions{ {Namespace: epSlice1.Namespace, Name: "svc1"}: { epSlice1.Name: epSlice1.ResourceVersion, }, }, </ins> }, "different service": { updateParam: epSlice1, checksParam: epSlice1DifferentService, expectHas:   false, expectStale: true, <ins> expectResourceVersionsByService: map[types.NamespacedName]endpointSliceResourceVersions{ {Namespace: epSlice1.Namespace, Name: "svc1"}: { epSlice1.Name: epSlice1.ResourceVersion, }, }, </ins> }, "different resource version": { updateParam: epSlice1, checksParam: epSlice1DifferentRV, expectHas:   true, expectStale: true, <ins> expectResourceVersionsByService: map[types.NamespacedName]endpointSliceResourceVersions{ {Namespace: epSlice1.Namespace, Name: "svc1"}: { epSlice1.Name: epSlice1.ResourceVersion, }, }, </ins> }, }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The endpointSliceTracker creates a set of resource versions for each service, the resource versions in the set could be deleted when endpointslices are deleted, but the set and its key in the map is never deleted, leading to memory leak. What you expected to happen: The set of resource versions should be delete from endpointSliceTracker when the service is deleted. Environment: Kubernetes version (use ): 1.18.5
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: rs.Status.AvailableReplicas == newStatus.AvailableReplicas && rs.Generation == rs.Status.ObservedGeneration && reflect.DeepEqual(rs.Status.Conditions, newStatus.Conditions) { <del> return nil </del> <ins> return rs, nil </ins> } // deep copy to avoid mutation now. // TODO this method need some work.  Retry on conflict probably, though I suspect this is stomping status to something it probably shouldn't copyObj, err := api.Scheme.DeepCopy(rs) if err != nil { <del> return err </del> <ins> return nil, err </ins> } rs = copyObj.(*extensions.ReplicaSet)

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? dangling ReplicaSet Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure A1 OS (e.g. from /etc/os-release): NAME="Container Linux by CoreOS" ID=coreos VERSION=1235.9.0 VERSIONID=1235.9.0 BUILDID=2017-02-02-0235 PRETTYNAME="Container Linux by CoreOS 1235.9.0 (Ladybug)" ANSICOLOR="38;5;75" HOMEURL="" BUGREPORTURL="" Kernel (e.g. ): Linux node-3-vm 4.7.3-coreos-r2 SMP Thu Feb 2 02:26:10 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools: own Others: What happened: Created new deployment by altering the image tag of the deployment descriptor and using kubectl apply -f "". Noticed that one container of the previous deployment stayed there. What you expected to happen: All containers from the previous deployment should have been terminated. How to reproduce it (as minimally and precisely as possible): I cannot reliably trigger this, I do several deployments per hour and sometimes this problem manifests itself. Anything else we need to know: Deployment descriptor:
Can you also post the output of kubectl get deployment tviftttapp -o yaml kubectl get rs (dangling rs) -o yaml When this manifests again?
Here are the current deployment, the dangling RS, and the "good" RS. Deployment: Dangling RS: Current RS:
It seems that status.availableReplicas for the new replica set is never correctly updated to 2. Short term if you stop using minReadySeconds (minReadySeconds=2 doesn't offer you much anyway), this problem should stop. Long term, we should fix the status update. cc:
can one of you guys paste logs from the controller manager when this issue occurs? Log level should be ideally set to 2.

I managed to reproduce it:
Will close this one once is cherry-picked into 1.5
4 is out and contains a fix that should force the Deployment to be resynced eventually and have availableReplicas updated. Not optimal yet but eventually consistent. I have queued up some follow-up work to refine the resync.
Does anyone know when version 1.5.4 OR 1.6 will be available using Redhat package downloads (in RPM form) ? I only see version 1.5.2 being the latest there. I have the same problem that is being discussed in this thread and want to upgrade so that my K8S cluster can support Deployments.


Question: Type=oneshot RemainAfterExit=yes ExecStartPre=/usr/bin/mkdir -p /opt/cni <del> ExecStartPre=/usr/bin/curl --fail --silent --show-error --location --create-dirs --output /opt/downloads/cni.tar.gz https://storage.googleapis.com/kubernetes-release/network-plugins/cni-9d5e6e60e79491207834ae8439e80c943db65a69.tar.gz </del> <ins> ExecStartPre=/usr/bin/curl --fail --silent --show-error --location --create-dirs --output /opt/downloads/cni.tar.gz https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz </ins> ExecStart=/usr/bin/tar xf /opt/downloads/cni.tar.gz -C /opt/cni/ - name: kubernetes-install-rkt.service

I have a container that got the broadcast address of podCIDR subnet. My CNI conf is: And the container in question has which is the broadcast address for the subnet. So it can't figure out the MAC of cbr0 because all frames are getting sent to (or at least, that's what seems to indicate). Eg: tcpdump alonside a wget on vs a similar normal container
Looks like a bug in ipam. Will try to put together a fix.
I stress tested your fix overnight with the networking suite I'm writing (the one that found the bug), didn't hit the failure in 100 runs, we should get it in asap
Great. It would be better to pull from upstream once the fix is merged. Otherwise, we have to fork it like we did in 1.3.
Marking this p0 flake so it shows up in release queries
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: "TEST_ETCD_VERSION="+TestContext.EtcdUpgradeVersion, "STORAGE_BACKEND="+TestContext.EtcdUpgradeStorage, "TEST_ETCD_IMAGE=3.1.10") <ins> } else { // In e2e tests, we skip the confirmation prompt about // implicit etcd upgrades to simulate the user entering "y". env = append(env, "TEST_ALLOW_IMPLICIT_ETCD_UPGRADE=true") </ins> } v := "v" + rawV

/kind bug /priority failing-test /sig cluster-lifecycle testing This test fails: https://k8s- Some clues: waiting for new master timeout: master healthz check failed: etcd is crashloopbacking: I guess the cause may be something associated with etcd version compatibility. any thoughts or suggestions? Thanks! This is also posted in !topic/kubernetes-sig-cluster-lifecycle/c4MW3R5v4v0 /cc
xref: /cc
cc
cc From my understanding, this is expected as etcd can't handle downgrades. I think this has been worked around though in previous releases by avoiding to downgrade etcd on k8s downgrade. That's what we're gonna do in kubeadm at least
Yes, per the etcd , "[automatic] downgrade from this completed state is not possible... Please backup the data directory of all etcd members to make downgrading the cluster possible even after it has been completely upgraded."
thanks! Is there any way to configure the downgrade test not to downgrade etcd? However, this might cause another problem: if we don't downgrade etcd from 3.1.10 to 3.0.17 when downgrading cluster from 1.9 to 1.8, can the 1.8 cluster work well against 3.1.10 etcd since the officially supported etcd version of 1.8 is 3.0.17?
great question. We would probably need to setup tests to actually test it, but I really believe that it should work fine (k8s 1.8 has already etcd client in 3.1.10 version). And also, we don't really have big choice here - that's the only thing we can do/recommend doing.
reasonable. Totally agreed :) So now the concern here is that how we should configure the downgrade test not to downgrade etcd to avoid the job failing. any ideas on the config of this job?
cc
I'm pretty sure we used to do that in the past (when we were doing etcd 2.2.1 -etcd 3.0.17 upgrade in 1.5 -1.6 release (though those might have been manual tests).

Assuming that the tests are setup on GCE, it should be enough to set some env variables for the downgrade to pin the downgrade to a particular release: ETCDIMAGE and ETCDVERSION seem to be those ones.
thanks for the info! I will send a PR (touching some e2e/cluster-lifecycle) trying to specify the etcd version 3.1.10 when the cluster downgrade target version is 1.8 but I'm not sure if it's a good solution for this issue.
FYI I'm proposing to unblock the tests from running. A better long-term fix is needed.
I send another workaround I guess it won't conflict with your PR. It may be a starting of a long-term fix.
We need a good release note to warn users that they need to prevent an etcd downgrade if they downgrade k8s. We filled in the warning from 1.6 as a placeholder, but it needs to be updated for this specific issue: We also should add a warning to the upgrade script, similar to this one from 1.6: cc who were involved with the update to 3.1.10 () Can anyone advise on the above?
Looks like we need something along the lines of this for : I'll fill out the google doc for the release notes as well.
The idea you mentioned of checking for and env vars before emitting a warning in sounded useful. is the docker tag of the for image to run etcd from. Each image contains multiple etcd binaries. For example: determine which etcd binary is run. It is passed to as which is used to select the exact etcd binary. If I understood your suggestion right, we could check if and are set to before emitting a warning in and skip the warning if they are set to 3.0.x, since an and etcd upgrade will not occur.
I manually verified that the following works as expected to downgrade from 1.9 to 1.8: Before upgrading, my 1.8 cluster had etcd 3.0.17. After upgrading, my 1.9 cluster had etcd 3.1.10. After downgrading, my 1.8 cluster had etcd 3.1.10 still. I'll work on adding a prompt to if these are unset.

[MILESTONENOTIFIER] Milestone Issue Current Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
The latest downgrade test is running with the config by : Note the But the test still seems to stuck on waiting for new master: . There is no more log so I'm not sure if it's still due to etcd.
also xref
thanks! I didn't notice that comment :)
Let's wait for the next run against :) Is there anyway to stop the current stuck run? It lasts so long (~15h) for each run.
you should be able to reference the variant, which runs more often
Which just had it's first green run. On Dec 12, 2017 19:20, "Sen Lu" <:


Question: if [[ "${enable_ip_alias}" == 'true' ]]; then ret="--network-interface" ret="${ret} network=${networkURL}" <del> # If address is omitted, instance will not receive an external IP. ret="${ret},address=${address:-}" </del> <ins> if [[ "${address:-}" == "no-address" ]]; then ret="${ret},no-address" else ret="${ret},address=${address:-}" fi </ins> ret="${ret},subnet=${subnetURL}" ret="${ret},aliases=pods-default:${alias_size}" ret="${ret} --no-can-ip-forward"

Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: serviceChanges:        proxy.NewServiceChangeTracker(newServiceInfo, ipFamily, recorder, nil), endpointsMap:          make(proxy.EndpointsMap), endpointsChanges:      proxy.NewEndpointChangeTracker(hostname, nil, ipFamily, recorder, nil), <ins> initialSync:           true, </ins> syncPeriod:            syncPeriod, minSyncPeriod:         minSyncPeriod, excludeCIDRs:          parsedExcludeCIDRs,

Entry in ipvs real sever remains at weight=0 after kube-proxy restarts. (During 1.21 -1.22 kubernetes cluster upgrade.) Related to As the related issue is considered closed, I will report this issue again as the problem still persists in newer version. Might because of some corner cases. Real server weight should be 1 when endpoints is ready. Use kube-proxy in ipvs mode Deploy a service resource and its backend pods Make real sever not ready (or may need node not ready) so as to trigger graceful delete in kube-proxy Before kube-proxy deletes the entry from ipvs, restart kube-proxy Kube-proxy becomes ready and leave the weight=0 entry as it is Service is no longer accessible through cluster ip When accessed, dmesg shows kernel messge error ofDeletion of pod will trigger kube-proxy to remove weight=0 entry from ipvs so that after the entry is recreated service will become accessible If service is 1. LoadBalancer, and, 2. .spec.externalTrafficPolicy: Local, and, 3, service is accessed from loadbalancer IP, the entry in ipvs for loadbalancer IP will only have endpoints in the same node. If backends is DaemonSet, the only one endpoints will have weight=0 thus causing connection failure After kube-proxy restarts, endpoints will be processed in the following lines. Only when the entry is in the gracefuldeleteManager.InTerminationList will be deleted. Which is not the case if kube-proxy restarts before it deletes the ipvs entry. An existing entry in the ipvs is sufficient for endpoint to be considered healthy. Weight is not considered. kube-proxy should treat ipvs entry properly by the either the following two ways. kube-proxy should initiate InTerminationList and put ipvs entry with weight=0 into the termination list kube-proxy should also look for weight in the ipvs entry instead of only UniqueRSName <details</details<detailson-premises </details<details</details<details</details<details</details<details</details
/sig network
Graceful delete of ipvs is not only triggered when pod is in terminating phase. Graceful delete seems to be triggered when endpointslice status is ready=false. This will happen when node become not ready.

However, after the node become ready again, pods on the node will only be recreated if the duration of NotReady is longer than certain time. Otherwise the pod will just become ready again. If the node will not recreate the pods && kube-proxy set the pods to the graceful delete queue && kube-proxy restarts before deletion and all cache is cleared, kube-proxy will keep ipvs entry with weight=0 even if the real server is ready. Please correct me if I misunderstand something.
/area ipvs /area kube-proxy
Seems like a legit bug to me. I can't triage until wednesday though.
I can't recreate this when I try to follow thre instructions with pod not-ready rather than node not ready. When the pod is made ready again is not stated, so I tried; a connection to a pod it not-ready. The weight is set to 0, the connection prevails -start kube-proxy. The weight is still 0, connection still working the pod ready weight is set to 1 after a while, as it should
In the instruction is different, the pod is made ready while is away. That instruction shows the fault. The reproduce instruction in this issue is incorrect.
I triage this issue as a bug, but it's really a duplicate of which was never fixed (it seems). Don't know if the proper action is to close this issue and re-open . /triage accepted
Please instruct if this issue shall be closed as duplicate and re-opened PTAL at
Thank you for your triage Monitoring IPVS weights is costly, but needed while this problem persists. Any news? Adding the following lines should fix the problem but will also add loads.
That does seem costly to do on every endpoint sync. Is there an alternative solution we can explore where we assess all endpoints once when kube-proxy initially starts up?
This is a startup-only problem. It should be solved with a one-time action.
How about modifying to be ? We could then check and update the real server weights only when is true. Only and will be modified to call , all other users will call the function with false. Since these two functions, are only called during startup, we won't be doing this monitoring every endpoint sync.
cc:


Question: import ( "context" "fmt" <ins> "os" </ins> "strconv" "strings"

What happened: Currently starting Kubectl failing on Azure Stack. What you expected to happen: Kubectl should start on Azure Stack environment. How to reproduce it (as minimally and precisely as possible): Try to deploy Kubernets on Azure Stack environment.. Anything else we need to know?: Environment: Kubernetes version (use kubectl version): master Cloud provider or hardware configuration: OS (e.g: cat /etc/os-release): Kernel (e.g. uname -a): Install tools: Others: /sig azure /assign
is it ? is a tool ...
my bad its kubelet. I update the title.
then could you provide the kubelet log? I am not sure kubelet on Azure stack is running as a container or process, you could get the logs by: And how do you deploy k8s on azure stack? cc


Question: if !reflect.DeepEqual(pip.Spec.Env, pod.Spec.Containers[0].Env) { framework.Failf("env of pod container does not match the env of the pip: expected %#v, got: %#v", pip.Spec.Env, pod.Spec.Containers[0].Env) } <ins> if !reflect.DeepEqual(pip.Spec.Env, pod.Spec.InitContainers[0].Env) { framework.Failf("env of pod init container does not match the env of the pip: expected %#v, got: %#v", pip.Spec.Env, pod.Spec.InitContainers[0].Env) } </ins> }) It("should not modify the pod on conflict", func() {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /sig cluster-lifecyle What happened: I created a Pod Preset to inject an environment variable. The Pod container gets properly injected, but the init container does not. A small sample is here: What you expected to happen: The init container would get the pod preset injected How to reproduce it (as minimally and precisely as possible): Run kubectl create -f on the following gist: Reproduced here: Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-09T07:26:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-10-17T15:09:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: minikube 0.23.0 OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: Started with:
/sig cluster-lifecyle /sig service-catalog
My use case: I have an a git init container that checkout out configuration. This needs to run before the main container (hence why it is an init container). I'd like to be able to simply annotate the pod with a label and have various environment variables (GIT_REPO, etc) supplied to the init container
Update from Slack discussion: Supporting Init containers is in the Beta proposal for PodPreset. PodPreset resource is going to be moved from core to service-catalog. We have an implementation based on Initializers, but looks like Initializers are not going to Beta soon and we might have to reimplement them using Webhooks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
It'd be nice if this worked, I suspect it would help the folks over at Knative Build implement support for passing HTTP_PROXY env vars and CA cert secrets to build steps.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a customer interested in seeing this fixed - if we submit a PR, are there objections to merging it?
cc
This is also affecting me - I have an init container that loads data from AWS Secrets and stores them in a volume that is accessed by the main container. The init container needs credentials that are passed via a podpreset
any ETA when support for initcontainers will be provided by PodPreset ?
This was fixed in 1.14.


Question: DOCKER_OPTS="" <del> {% if grains.docker_opts %} </del> <ins> {% if grains.docker_opts is defined and grains.docker_opts %} </ins> DOCKER_OPTS="${DOCKER_OPTS} {{grains.docker_opts}}" {% endif %} DOCKER_OPTS="${DOCKER_OPTS} --bridge cbr0 --iptables=false --ip-masq=false -r=false"

I get this from salt. I would fix it, but looks like this commit reversed what I would have done: Not sure why.
Darn it -- those guys said they tested it. I'll fix this up.
cc: -- FYI
Bah... was an attempt to fix this issue for a different case. I'll look at it again.
I'm running a couple of e2e tests on a fix now.
please reopen if still broken.


Question: sc               *storagev1.StorageClass pvc              *v1.PersistentVolumeClaim pv               *v1.PersistentVolume <ins> driverName       string </ins> testVolType      testpatterns.TestVolType nodeName         string volMode          v1.PersistentVolumeMode

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Probably caused by <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
https://k8s- /assign
I understand that moved crd creation logic to e2e test's k8s cluster deployment code. On the other hand, I tested it with manually deployed k8s cluster. So, in my case, I guess that in addition to enable and feature gates enabled, I need to create crd manually before tests, like below: I could make two of three SkipAttach tests pass in my environment by above way. For "non-attachable volume does not need VolumeAttachment" case, I'm still looking into it. (It doesn't seem to skip creating volumeAttachment, properly)
I found a typo in 's argument. While hostpath driver's connects driver name and unique name without "-", it connects them with "-". This makes "non-attachable" test fail, due to the mismatch between volume name in crd and provisioner name, as a result, creating volumeAttachment isn't skipped. I will create a PR to fix it. For above bug, is unrelated. It just changed the way in how to test in manually deployed k8s cluster as I mentioned in above comment. So, there should be no problem for the PR, at least for the CI testing if it works well in CI environments.
I can verify that the CRDs are not getting installed on the alpha cluster. Investigating why.
/reopen since I see the Alpha tests failing
/milestone v1.13 /priority important-soon /kind failing-tests
/kind failing-test
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is fully resolved. There are still failures on https://k8s- /reopen
Reopened this issue. <detailsIn response to : I don't think this is fully resolved. There are still failures on https://k8s- Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you help take a look at these failures? It's testing CSI filesystem volume mode. I think it is unrelated to the crd installation issue that Saad fixed.
I tested with AllAlpha flag and manually created csi crds and confirmed that filesystem volume mode test passed in my environment and also other tests passed there. If I delete csi crds, other tests also fail, so I agree that the issue that Saad fixed is not related. However, above also means that this issue can't be reproduced in my environment.
I1109 14:56:01.141] Nov 9 14:56:01.131: INFO: At 2018-11-09 14:50:54 +0000 UTC - event for security-context--e42e-11e8-9b95-: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc--e42e-11e8-880b-" : node "bootstrap-e2e-minion-group-3wtg" has no NodeID annotation
Hostpath also seems to be failing because of driver registration issues? Not sure why other alpha tests are passing if that's the case
This test is the only test that uses  to create pod with CSI driver volume and expects the result to be success. One of the pod parameters specified there might affect this failure.
Those 2 failing tests are newly introduced in and started to fail from the beginning. Is that PR blocking 1.13? If not, can we revert and fix later? I have recently set up optional presubmit tests for alpha features at , which will pick up those 2 tests. Currently this failure is blocking
apart from the 2 failing tests, I see which addressed with is also flaky. These failures are now blocking PRs as mentioned above. As we enter code slush today, its very critical to have the PRs get in sooner than later. Can we plz revert the feature and the tests until they are fully resolved and ready to go. Thanks
I prefer disabling the specific tests instead of reverting the entire pr. We will lose important signal for csi
That works for me.

Can you cc me on the PR that disables those tests? When you re-enable the tests, remember to try them out on . :)
Let me consider if we can just skip these test for CSI, instead of reverting the entire patch.
you mean just skipping them in Alpha cluster right?
I was thinking about skipping the below test with the combination of below drivers: [Test] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode[Feature:BlockVolume] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources [Driver] csi-hostpath Won't it enough? Or, do we need to skip it only with alpha cluster?
If we just disable the tests alone and not revert the feature, we are just masking the issue. Ideally if we are not able to fix the tests and reenable them the feature should be reverted from 1.13 as well. Are you also actively working on fixing the tests and this skip is just to unblock PRs in the meantime? if so can we timebox by when we can have the tests fixed and re-enabled / the feature reverted? My fear is we might forget the fact that its disabled since there's no active reporting on it.
Actually, there is no feature in either of culprit commits. It tests or intended to improve the way it is tested. Therefore, it must be a test issue or another issue which is found by the test , not an issue introduced by the commits themselves. In the code, there is a clear comment that the test is skipped due to this issue with the reference to this issue#, so this won't be forgot. If you still worry about this issue is not tracked, we might be able to keep this issue open.
IIUC the test that were failing (and are being reverted) were for the new CSI festures going to Alpha in 1.13, right? so even these 2 culprit commits do not contain feature work, it is testing some new code additions to 1.13 and now those will be untested until we re-enable these tests. Is there an ETA to re-enable them, preferably sometime early next week? Yes +1 to keeping this issue open to track the skipped tests.
The two failing tests themselves are not for new alpha features that are in v1.13.

Instead, they are just to test csi driver works well as it works in existing in-tree driver. (And failed because they might be affected by some other alpha features or some of specific configurations in gce.) In addition, ~50 tests for csi driver, so it would be better not to remove all of the tests by reverting the PR, just for the 2 failing tests. I think that just reverting the patch won't help improving quality of k8s. As for ETA to solve the issue, I found difficulty in debuging in my local environment. It seems that the two tests fail in gce environment w/ AllAlpha=true, however, as I've already mentioned, at least csi-hostpath tests in my local environment w/ AllAlpha=true doesn't fail.(And I don't have test environment for gce.) Of course, I will help solving the issue next week. However, I will need someone's help for testing combinations of alpha features in gce environment to narrow down which flag really affects this issue. At least I would like to know the results: w/ MountPropagation=true,CSIPersistentVolume=true,BlockVolume=true,CSIBlockVolume=true all alpha flags except CSIDriverRegistry and CSINodeInfo
/reopen Thanks for the detailed explanation. I think the revert is fine in this case then. I am keeping this issue open however to track and fix the failures in these 2 tests can any of you help answer w.r.t alpha features on gce environment. thanks
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can help debug when I'm back on Tuesday. In the meantime, I would suggest looking through the failing test logs to try to triage what's going on
Here's one failing run:
I looked into the csi-hostpath failure log. As seen in the for , where the pod which has a csi-hostpath volume is being attached is scheduled, mount volume fails with these errors (1). Related code to this log is (2), so the driver doesn't seem to be registered to . As for the registration code for csiDriver, it is called as (3).

Therefore, will be logged if the driver is triggered for registration. As seen (4), registration is only triggered in , which is different from the one the pod is scheduled. In addition, driver-registrar seems to be deployed only on the node where registration is triggered (5). Is it right behaviour, while registrar is deployed as daemonset? I guess that just this fact seems to cause an issue to make csi-hostpath driver work properly. (Actually, I only tested with 1 node environment, so this might only happen in multiple nodes environment.) <details- (1) (2) Result of grep to check whether registrar is deployed to each node [ for bootstrap-e2e-minion-group-50r7][ for bootstrap-e2e-minion-group-k781][ for bootstrap-e2e-minion-group-z5cg] </details
The hostpath plugin should be deployed as a DaemonSet, so it's strange that it's only getting registered on only one node.
/assign looking into this as well
could you link the PR for skipping [external-gcepd] tests into this issue when you do that as well
skipping gcePD-external is here:
Since gce-cos-alphafeatures is now passing, do we need to keep this issue open? Should it morph into a non-blocking issue for longterm work on a better test?
We have root caused 2 of the 3 failures, which were problems in the test and test conifg. Fixes will be out soon. Still triaging the 3rd failure, which could be a real bug in alpha clusters
I've root caused csi-hostpath failure issue. It just failed to set the node properly in test code side. I will make a PR to fix it.
Or should it be done in your PR too, to avoid modifying twice in the separate PR? Actually, the fix are as below: (Failing to set node before pod creation)
Created for csi-hostpath fix, to prepare for the case that we need a separate PR. We can discuss more on this there, if needed.
/unassign found possible root cause for GCE issue: CSI NodeID annotation is not present on the Node object, could be caused by failed retries on updating Node object.

(apparently when alpha features are open we shouldn't be looking at the Node object anyway except we seem to be, so that might also be a bug). has the most context for the fix /assign
The symptom of the remaining test failure is the test pod fails to create because it fails to attach with the error: . The test succeeds the first time it's run in a cluster, and always fails after that. Two issues caused this error message. the CSI external-attacher tries to fetch the object to get the NodeID, it does so through an informer. The first time the test is run, objects are created for the very first time, and the informer returns the correct object. In subsequent test runs, s from the first run is kept, and for some reason the informer fails to detect the object and returns empty. Making the API call directly works, I'll patch the external-provisioner with this to mitigate the issue, and this is sufficient to get the test to pass. The informer bug will need to be eventually fixed. fails to post the NodeID annotation sometimes, likely due to node status patch not being retried correctly. This is crucial for clusters without feature flag set, and this is the default for 1.13. Will add the appropriate retry mechanism.
Opened to track the informer issue
Are and potential PR for the only 2 fixes being actively worked on for this issue?
Only is needed. We have a workaround for
will you have a followup PR to enable the CSI alpha tests back? or is that already in flight?
also reenables the tests
OK with merged we should see the CSI alpha tests re-enable. to keep an eye out for CI signal
The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! /close
Closing this issue. <detailsIn response to : The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The tests are actually running (and passing) in the normal suite now, now that the raw block feature moved to beta.


Question: "k8s.io/component-base/logs" "k8s.io/kubernetes/cmd/cloud-controller-manager/app" <del> _ "k8s.io/component-base/metrics/prometheus/version" // for version metric registration // NOTE: Importing all in-tree cloud-providers is not required when // implementing an out-of-tree cloud-provider. </del> _ "k8s.io/component-base/metrics/prometheus/clientgo" // load all the prometheus client-go plugins <del> _ "k8s.io/legacy-cloud-providers/aws" _ "k8s.io/legacy-cloud-providers/azure" _ "k8s.io/legacy-cloud-providers/gce" _ "k8s.io/legacy-cloud-providers/openstack" _ "k8s.io/legacy-cloud-providers/vsphere" </del> <ins> _ "k8s.io/component-base/metrics/prometheus/version"  // for version metric registration </ins> ) func main() {

make verify WHAT=typecheck-providerless outputs /cc /milestone v1.19
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
see example successful run:
For reference -
fix in
screaming


Question: trace.Step("Basic checks done") startPredicateEvalTime := time.Now() <del> filteredNodes, failedPredicateMap, err := g.findNodesThatFit(pluginContext, pod, nodes) </del> <ins> filteredNodes, failedPredicateMap, err := g.findNodesThatFit(pluginContext, pod, nodeLister) </ins> if err != nil { return result, err }

In : It seems that nodes are only used in the case there are no predicates. This call can be moved inside . Moving it there saves us from creating the node slice unless it is necessary. Checking if there are no nodes (line 202 below) can be done using . /kind cleanup /sig scheduling
I can work on this


Question: prenormalizedDesiredReplicas: 5, expectedStabilizedReplicas:   5, expectedRecommendations: []timestampedRecommendation{ <del> {5, time.Now()}, </del> <ins> {5, now}, </ins> }, }, { name: "simple scale down stabilization", key:  "", recommendations: []timestampedRecommendation{ <del> {4, time.Now().Add(-2 * time.Minute)}, {5, time.Now().Add(-1 * time.Minute)}}, </del> <ins> {4, now.Add(-2 * time.Minute)}, {5, now.Add(-1 * time.Minute)}}, </ins> currentReplicas:              100, prenormalizedDesiredReplicas: 3, expectedStabilizedReplicas:   5, expectedRecommendations: []timestampedRecommendation{ <del> {4, time.Now()}, {5, time.Now()}, {3, time.Now()}, </del> <ins> {4, now}, {5, now}, {3, now}, </ins> }, scaleDownStabilizationWindowSeconds: 60 * 3, },

When configuring new behaviors section in HPA I two scenarios that I think are incorrect. First one: stabilizationWindowSeconds set up to 5 minutes for both scale up and scale down min replica count 2 Application consumes big amount of resources during warm-up 0 minute: current replica count: 2 1 minute: desired replica count: 3 - no scale up stabilization window worked 2 minute: desired replica count: 1 (!) - scale up to 3 (max number from previous recommendations during stabilization window period) ! Second one Stabilization window for scale up 0 Stabilization window for scale down 5 minutes min replica count 2 behavior set to scale up new pod in 2 minutes scaleUp: policies: type: Pods value: 1 periodSeconds: 120 What happens: 0 minute: 2 replicas 1 minute : performance spike scale up to 3 (desired value 4) 2 minute : no spike no scale up 3 minute: no spike scale up to 4(!) - supposedly desired value 1 and supposedly worked this way because of scale down window set to 5 minutes picking max value from that 5 minutes window that was 4 replicas Expected behaviors 1 Scenario no scale up for 5 minutes 2 Scenario correct measurement and no scale up when there is no need for it Environment: Kubernetes version (use ): client 1.19.3 Server 1.19.0 Cloud provider or hardware configuration: Azure aks
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
/assign josephburnett
The HPA controller keeps a flat history of recommendations for stabilization. ! When downscale stabilizing the maximum recommendation is selected. Purple is pre-stabilized recommendation. Green is final recommendation. Blue is max recommendation within window. ! When upscale stabilization the minimum recommendation is selected. ! However when both up and down scale stabilization is configured, the interpretation of the history changes depending on the direction of movement.

Red is the undesired, actuated recommendation. ! What we want is to keep the stabilized recommendation within the envelope of the minimum and maximum over configured stabilization windows. We should only move when the envelope forces a move. ! We first stabilize up and down, and check if the current value fits within it. If it does (or the range is "inverted"), we do nothing. If it does not, we emit the "nearest possible" value, which will be lower or upper bound, depending on where the current size is in relation to the range. (thanks and for the paragraph above)


Question: } // Validate validates the provided options <del> func (o *Options) Validate() error { </del> <ins> func (o *Options) Validate(args []string) error { if len(args) != 0 { return errors.New(fmt.Sprintf("extra arguments: %v", args)) } </ins> if o.Output != "" && o.Output != "yaml" && o.Output != "json" { return errors.New(`--output must be 'yaml' or 'json'`) }

ran instead of and got no error and then though that was the right version kubectl should fail for unexpected extra arguments
/sig cli
/assign
/triage accepted


Question: framework.Failf("Timed out after %v waiting for stable cluster.", timeout) break } <del> allScheduledPods, allNotScheduledPods = getFilteredPods(c, masterNodes, metav1.NamespaceAll) </del> <ins> allScheduledPods, allNotScheduledPods = getScheduledAndUnscheduledPods(c, masterNodes, metav1.NamespaceAll) </ins> } return len(allScheduledPods) } <del> // getFilteredPods lists scheduled and not scheduled pods in the given namespace, with succeeded and failed pods filtered out. func getFilteredPods(c clientset.Interface, masterNodes sets.String, ns string) (scheduledPods, notScheduledPods []v1.Pod) { </del> <ins> // WaitForPodsToBeDeleted waits until pods that are terminating to get deleted. func WaitForPodsToBeDeleted(c clientset.Interface) { startTime := time.Now() deleting := getDeletingPods(c, metav1.NamespaceAll) for len(deleting) != 0 { if startTime.Add(timeout).Before(time.Now()) { framework.Logf("Pods still not deleted") for _, p := range deleting { framework.Logf("%v/%v", p.Namespace, p.Name) } framework.Failf("Timed out after %v waiting for pods to be deleted", timeout) break } time.Sleep(waitTime) deleting = getDeletingPods(c, metav1.NamespaceAll) } } // getScheduledAndUnscheduledPods lists scheduled and not scheduled pods in the given namespace, with succeeded and failed pods filtered out. func getScheduledAndUnscheduledPods(c clientset.Interface, masterNodes sets.String, ns string) (scheduledPods, notScheduledPods []v1.Pod) { </ins> pods, err := c.CoreV1().Pods(ns).List(metav1.ListOptions{}) <del> framework.ExpectNoError(err, "listing all pods in kube-system namespace while waiting for stable cluster") </del> <ins> framework.ExpectNoError(err, fmt.Sprintf("listing all pods in namespace %q while waiting for stable cluster", ns)) </ins> // API server returns also Pods that succeeded. We need to filter them out. filteredPods := make([]v1.Pod, 0, len(pods.Items)) <del> for _, pod := range pods.Items { if pod.Status.Phase != v1.PodSucceeded && pod.Status.Phase != v1.PodFailed { filteredPods = append(filteredPods, pod) </del> <ins> for _, p := range pods.Items { if !podTerminated(p) { filteredPods = append(filteredPods, p) </ins> } } pods.Items = filteredPods return e2epod.GetPodsScheduled(masterNodes, pods) } <ins> // getDeletingPods returns whether there are any pods marked for deletion. func getDeletingPods(c clientset.Interface, ns string) []v1.Pod { pods, err := c.CoreV1().Pods(ns).List(metav1.ListOptions{}) framework.ExpectNoError(err, fmt.Sprintf("listing all pods in namespace %q while waiting for pods to terminate", ns)) var deleting []v1.Pod for _, p := range pods.Items { if p.ObjectMeta.DeletionTimestamp != nil && !podTerminated(p) { deleting = append(deleting, p) } } return deleting } func podTerminated(p v1.Pod) bool { return p.Status.Phase == v1.PodSucceeded || p.Status.Phase == v1.PodFailed } </ins>

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: soak-gce-gci-1.13, others Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates MaxPods limit number of pods that are allowed to run [Slow] Since when has it been failing: Rougly forever. shows how this test was flaking back during the 1.11 cycle. There's been no sudden increase in frequency, but a test that flakes roughly 1.5 times per hour needs looking into, and probably tweaking. Testgrid link: https://k8s- Reason for failure: Most of the time, these flakes are timeouts. However, sometimes they have other causes; see Triage link: Anything else we need to know: This is part of a general effort to clean up our CI signal by eliminating longstanding, frequent flakes. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /priority important-longterm /sig scheduling attn:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /help-wanted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
There are several different failures in this test, I was in particular looking at which usually boils down to something like this My hunch is that we are trying to schedule more than we should. This is how we compute how many pods to schedule which just takes max capacity and substracts whatever this returns My guess would be that the computation is not completely correct. My idea would be to add more logging into to be able to figure out more. Does that sound reasonable to you or would you take another approach? cc.

I went over all the pods that were on the node before the test started and then after and diff-ed all the pods that are not started by the test "grep -v maxp-" and did not found any pod that would start in the middle of test and was not present in the beginning :/ but there were 18 pods in the beginning, 330 is the full capacity so we should have started 312 to fully saturate, while we were trying to start 313. (maybe some pod restarted right when we were starting the test?) I am looking at this latest case but there's many like this one
Can we possibly be missing some pods here? pods with no node assigned that are not unschedulable?
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This just got much worse in the last weeks, I tried looking into PRs along the time and hit into this could someone please verify it that could have made things worse? I'll also try to look tomorrow since I investigated this while ago :)
Anyone can help here? It's constantly red for several days, it's not even flaking anymore in our release dashboards /priority important-soon


Question: import ( "fmt" <ins> "net" </ins> "strings" "testing"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There is a bug in ipv6, see: What you expected to happen: The judgement logic should be: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network /area ipv6
/assign cc


Question: if err != nil { return nil, fmt.Errorf("failed to get region from zones: %v", err) } <del> volID = fmt.Sprintf(volIDZonalFmt, UnspecifiedValue, region, pv.Spec.GCEPersistentDisk.PDName) </del> <ins> volID = fmt.Sprintf(volIDRegionalFmt, UnspecifiedValue, region, pv.Spec.GCEPersistentDisk.PDName) </ins> } else { // Unspecified volID = fmt.Sprintf(volIDZonalFmt, UnspecifiedValue, UnspecifiedValue, pv.Spec.GCEPersistentDisk.PDName)

Was just perusing the code and looks like we translate regional disks into the "zonal volume ID fmt", and there doesn't seem to be any tests for regional disks running so we should confirm that this works and fix it if it doesn't /assign /cc
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /assign
/assign


Question: "STANDARD_D4_V3":         8, "STANDARD_D5_V2":         64, "STANDARD_D5_V2_PROMO":   64, <ins> "STANDARD_D64AS_V4":      32, "STANDARD_D64A_V4":       32, </ins> "STANDARD_D64S_V3":       32, "STANDARD_D64_V3":        32, "STANDARD_D8AS_V4":       16, "STANDARD_D8A_V4":        16, "STANDARD_D8S_V3":        16, "STANDARD_D8_V3":         16, <ins> "STANDARD_D96AS_V4":      32, "STANDARD_D96A_V4":       32, </ins> "STANDARD_DC1S_V2":       1, "STANDARD_DC2S":          2, "STANDARD_DC2S_V2":       2, "STANDARD_DC4S":          4, "STANDARD_DC4S_V2":       4, <del> "STANDARD_DC8S":          8, </del> "STANDARD_DC8_V2":        8, "STANDARD_DS11-1_V2":     8, "STANDARD_DS11":          8,

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: }) } <ins> func TestSimpleDaemonSetRestartsPodsOnTerminalPhase(t *testing.T) { for _, podPhase := range []v1.PodPhase{v1.PodSucceeded, v1.PodFailed} { t.Run(string(podPhase), func(tt *testing.T) { forEachStrategy(tt, func(t *testing.T, strategy *apps.DaemonSetUpdateStrategy) { ctx, closeFn, dc, informers, clientset := setup(t) defer closeFn() ns := framework.CreateNamespaceOrDie(clientset, "daemonset-restart-terminal-pod-test", t) defer framework.DeleteNamespaceOrDie(clientset, ns, t) dsClient := clientset.AppsV1().DaemonSets(ns.Name) podClient := clientset.CoreV1().Pods(ns.Name) nodeClient := clientset.CoreV1().Nodes() podInformer := informers.Core().V1().Pods().Informer() informers.Start(ctx.Done()) go dc.Run(ctx, 2) ds := newDaemonSet("restart-terminal-pod", ns.Name) ds.Spec.UpdateStrategy = *strategy if _, err := dsClient.Create(ctx, ds, metav1.CreateOptions{}); err != nil { t.Fatalf("Failed to create DaemonSet: %v", err) } defer cleanupDaemonSets(t, clientset, ds) numNodes := 3 addNodes(nodeClient, 0, numNodes, nil, t) validateDaemonSetPodsAndMarkReady(podClient, podInformer, numNodes, t) validateDaemonSetStatus(dsClient, ds.Name, int32(numNodes), t) podToMarkAsTerminal := podInformer.GetIndexer().List()[0].(*v1.Pod) podCopy := podToMarkAsTerminal.DeepCopy() podCopy.Status.Phase = podPhase if _, err := podClient.UpdateStatus(ctx, podCopy, metav1.UpdateOptions{}); err != nil { t.Fatalf("Failed to mark the pod as terminal with phase: %v. Error: %v", podPhase, err) } // verify all pods are active. They either continue Running or are Pending after restart validateDaemonSetPodsActive(podClient, podInformer, numNodes, t) validateDaemonSetPodsAndMarkReady(podClient, podInformer, numNodes, t) validateDaemonSetStatus(dsClient, ds.Name, int32(numNodes), t) }) }) } } </ins> func TestDaemonSetWithNodeSelectorLaunchesPods(t *testing.T) { forEachStrategy(t, func(t *testing.T, strategy *apps.DaemonSetUpdateStrategy) { ctx, closeFn, dc, informers, clientset := setup(t)

I can reproduce this reliably with Calico v3.25.0 CNI, installed straight from the upstream . It seems to be a regression in Kubernetes 1.27.0-rc.0 compared to 1.27.0-beta.0, at least I can't reproduce it with beta.0. Cluster is deployed, and everything is ready & healthy. After a series of node reboots some pods are stuck in state and never recover: This of course disrupts the CNI and affects the scheduled workloads. The daemonset is "happy" with 2 pods being out: If I delete the pod manually, the pod recovers without any issues. The pods are restarted. Bring up a cluster with graceful node shutdown, reboot nodes a few times (including multiple node at anoce). Additional detailed output: <details</details<details</details<detailsnone </details<detailsTalos Linux 1.4.0-alpha.3 </details<details</details<details</details<detailsCalico v3.25.0, also reproducible with 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I wonder if that is related to the change
cc this might a regression in 1.27.0-rc.0 with this PR
Even if it is, it sounds like a bug in daemonset instead. Why wouldn't it replace pods that are terminated? It can probably be reproduced in integration.
I think there might be something with the fact that Calico terminates with exit code '0', so it feels like a "successful" run. Still such pod should be restarted, so I'm not sure why it doesn't get restarted after a reboot.
let me ask to get more details about reproducing this locally. Are you running this on Kind? How do you trigger the the restarts? Can you share mode details on the node configuration, such as the graceful shutdown period?

I initially hit this problem under AWS, but then I narrowed it down to Calico CNI. I'm using Talos Linux, this is the full kubelet config, using 30s/10s graceful shutdown period: <details</detailsLet me try to build a reproduce instruction in the next comment.
I also have complete output of under the details in the issue itself.
The daemonset controller should still recreate the pod in that scenario. /sig apps
The reproducer is to follow the , but it should be same under any Talos platforms except Docker (which doesn't have a graceful node shutdown, and any kind of shutdown). I verified it with the assets from the latest . The smallest I can reproduce it with is 1+1 cluster, but it looks same way e.g. in 3+2 cluster: Once the cluster is up, all the pods are running & healthy (). If I issue a reboot of the worker node with , with shows that pod on the worker node quickly enters state, and never recovers.
At the same time I don't quite understand what is exactly wrong, as if I just do a daemonset with command, the pod also enters state, but it gets quickly restarted by the via transition.
I can also send kubelet logs, and anything else which might be helpful to debug the issue, I'm also open to do a debug session with screen sharing.
This is because of . The pod doesn't reach the phase in this case. The kubelet just recreates the containers. However, when a pod is actually marked as "Succeeded" then kubelet would stop looking at it. However, it should be the responsibility of the daemonset controller to recreate a pod.
Following the sig-apps advise opened the PR () with an integration test for DaemonSet. It passes for Failed pods, but fails for Succeeded. In other words, the succeeded pods are not restarted. I'm yet to investigate (probably tomorrow) why this is so, but it seems like a bug. In the meanwhile some expertise about DaemonSets would be helpful.
Thanks for reporting this. I suppose this might be related to the fact that pods (created by daemonset) can now be marked as in v1.27.

When graceful node shutdown happens, we terminate the pod with : which will trigger kubelet to terminate the pod in pod worker () and eventually I assume set a terminal phase on the pod. We should verify if that is what is happening here. In either case, +1 on comment that I would expect that daemonset controller should recreate a pod if it is marked completed.
I will note that the intent for graceful shutdown was always that pods could be marked Succeeded (vs just Failed), since the pod is indeed terminal. It's likely that this was previously racy (no guarantee succeeded made it to the apiserver) and fixed that. However, outside of graceful shutdown this should never occur - do we have any evidence a RestartAlways pod went succeeded in that case (wasn't clear from the thread)? That would be a serious bug.
I think Succeeded would also be possible in other scenarios, like deletion (deletionTimestamp!=nil). This I've tested for Job pods, but I think DS pods would behave the same. Except for deletion I think the behavior is limited to graceful node shutdown, which invokes the kill function, but respects the Succeeded phase: Other uses of the kill function unconditionally set Failed, such as the eviction manager: EDIT: oh, actually for Jobs we don't use restartPolicy=Always, but still deleted pods with restartPolicy=Always may still terminate with Succeeded phase as shown in this test:
Deletion with RestartAlways is ok, because the pod goes away reasonably fast (but we are adding latency in 1.27 to that path) as you note.


Question: framework.ExpectNoError(err, "while deleting") ginkgo.By("Checking CSI driver logs") <del> err = checkPodLogs(ctx, m.driver.GetCalls, pod, expectPodInfo, expectEphemeral, csiInlineVolumesEnabled, false, expectedNumNodePublish) </del> <ins> err = checkNodePublishVolume(ctx, m.driver.GetCalls, pod, expectPodInfo, expectEphemeral, csiInlineVolumesEnabled, false) </ins> framework.ExpectNoError(err) })

1 ci-kubernetes-e2e-gci-gce 1 ci-cos-containerd-e2e-cos-gce in the master blocking test grid 1 ci-kubernetes-e2e-gci-gce-network-proxy-grpc 1 ci-kubernetes-e2e-gci-gce-proto Kubernetes e2e suite.[It] [sig-storage] CSI Mock workload info CSI PodInfoOnMount Update should be passed when update from false to true 12/14 code thaw after 1.29.0 release was merged at the time and is related to PodInfoOnMount. /cc /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
testCase: podInfoOnMount should be passed when update from false to true kubelet log: csi-mockplugin-0-mock log: the function in the test case is not working as expected. expect log at Line 2, but actual log is at Line 3.
is called twice for the same volume sometime even though the volume is already published successfully. Not only this issue, but also other issues like I wonder if it is a potential bug or a normal behavior. cc what do you think?
good catch. I think that any test that depends on exact index of NodePublish call will be flaky, sometimes Kubelet has to retry NodePublish. It would be better to check for NodePublish parameter changes. For example, test should check there were 1 or more NodePublish calls with no pod info and then 1 or more NodePublish calls with pod info afterwards.


Question: "//pkg/api/legacyscheme:go_default_library", "//pkg/apis/core:go_default_library", "//pkg/capabilities:go_default_library", <del> "//pkg/client/chaosclient:go_default_library", </del> "//pkg/cloudprovider:go_default_library", "//pkg/cloudprovider/providers:go_default_library", "//pkg/credentialprovider:go_default_library",

/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?


Question: // An error will be returned if iptables fails to update or acquire the initial lock. // Once a proxier is created, it will keep iptables up to date in the background and // will not terminate if a particular iptables call fails. <del> func NewProxier(ipt utiliptables.Interface, sysctl utilsysctl.Interface, exec utilexec.Interface, syncPeriod time.Duration, minSyncPeriod time.Duration, masqueradeAll bool, masqueradeBit int, clusterCIDR string, hostname string, nodeIP net.IP) (*Proxier, error) { </del> <ins> func NewProxier(ipt utiliptables.Interface, sysctl utilsysctl.Interface, exec utilexec.Interface, syncPeriod time.Duration, minSyncPeriod time.Duration, masqueradeAll bool, masqueradeBit int, clusterCIDR string, hostname string, nodeIP net.IP, recorder record.EventRecorder, ) (*Proxier, error) { </ins> // check valid user input if minSyncPeriod > syncPeriod { return nil, fmt.Errorf("min-sync (%v) must be < sync(%v)", minSyncPeriod, syncPeriod)

When kube-proxy fails to open a nodeport, it gives up setting up the rest of the iptables rules for the service. Leaving the service not usable. Cannot even use it with clusterIP. Kube-proxy only logs this error in . No useful information can be retrieved with kubectl. Need to record a event at least.
Ideally would the changes be removed and then reported back? Is there a preferred log or place to return the error?
the event maybe a better solution; as other port maybe success.
Need to surface the problem somewhere. An event seems okay
, there's and in ; should we reuse it or create new recorders for ?
Just reuse it. After all, it is within the same component. Are you going to submit a PR?
sure, my pleasure.
Great! Thanks! Let me know when the PR is ready.


Question: setInitCommand(input.pod, fmt.Sprintf("ln -s /bin/notanexistingpath %s", input.subPathDir)) // Pod should fail <del> testPodFailSubpath(input.f, input.pod) </del> <ins> testPodFailSubpath(input.f, input.pod, false) </ins> }) It("should fail if subpath with backstepping is outside the volume [Slow]", func() {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The read only sig-storage tests validate that Pods can come up when readOnly is set. But it doesn't actually validate that the container cannot write to the volume. What you expected to happen: Tests should validate that the container cannot write to readonly volumes. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can this issue be closed after your last PR or is there more to work on? If yes I'd like to know what's left because I'm looking for something to work on
I think we should actually reinstate the test case I removed and enhance it to handle the two scenarios I mentioned. It would also be good to see if we have other readOnly test cases that could be enhanced as well
Thanks ! I have a few questions: 1) how do I run the tests in locally? and 2) how can I assert writing to volume has failed? Thank you in advance!!


Question: endpoint:             "/var/log/kubelet/plugins/myplugin/csi.sock", versions:             []string{"v1.2.3", "v0.3.0", "2.0.1"}, foundInDeprecatedDir: true, <del> shouldFail:           true, </del> <ins> shouldFail:           false, </ins> }, { pluginName:           "test.plugin", endpoint:             "/var/log/kubelet/plugins/myplugin/csi.sock", versions:             []string{"v0.3.0", "2.0.1"}, foundInDeprecatedDir: true, <del> shouldFail:           true, </del> <ins> shouldFail:           false, </ins> }, { pluginName:           "test.plugin",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: CSI allows 0.x drivers only in the deprecated kubelet plugin dir. CSI allows 1.x drivers only in the new kubelet plugin dir. If a driver implements both 0.x and 1.x (which is unlikely since the side car containers do not support this), kubelet requires the driver live in new kubelet plugin dir. So CSI drivers that implement both 0.x and 1.x if they want a seamless transition between k8s 1.12 to 1.13 would need to deploy in both old and new dir. What you expected to happen: argues that this is adding churn for CSI authors. He recommends allowing any driver that implements 0.x (even if it implements 1.x) to use either new or old (not just new). /milestone v1.13 If release team approves, PR is small and I can get prepare it pretty quickly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
discussed in the burndown meeting, please go ahead and prepare the PR
/sig storage /priority critical-urgent /milestone v1.13
we need to wait for a fix for this before lifting code freeze


Question: StabilityLevel: metrics.ALPHA, }, ) <ins> // CPUManagerPinningRequestsTotal tracks the number of times the pod spec will cause the cpu manager to pin cores CPUManagerPinningRequestsTotal = metrics.NewCounter( &metrics.CounterOpts{ Subsystem:      KubeletSubsystem, Name:           CPUManagerPinningRequestsTotalKey, Help:           "The number of cpu core allocations which required pinning.", StabilityLevel: metrics.ALPHA, }, ) // CPUManagerPinningErrorsTotal tracks the number of times the pod spec required the cpu manager to pin cores, but the allocation failed CPUManagerPinningErrorsTotal = metrics.NewCounter( &metrics.CounterOpts{ Subsystem:      KubeletSubsystem, Name:           CPUManagerPinningErrorsTotalKey, Help:           "The number of cpu core allocations which required pinning failed.", StabilityLevel: metrics.ALPHA, }, ) </ins> ) var registerMetrics sync.Once

Add to enable observability of the cpumanager. This is required to but it's also useful in general. To the best of my knowledge, it's very hard to observe the behavior of cpumanager without peeking into the pod and/or use the podresources interface (kubelet = 1.24). But even for the latter, no well known components expose metrics consuming this information. I believe anyway kubelet some expose some basic metrics about cpumanager behaviour, while deferring more fine grained metrics to external components (NFD, node_exporter, is fine)
/sig node /assign
/triage accepted /priority important-soon (as this is important for graduation for CPU Manager to GA)


Question: <del> apiVersion: v1 kind: ReplicationController metadata: name: dns-backend labels: name: dns-backend spec: replicas: 1 selector: name: dns-backend template: metadata: labels: name: dns-backend spec: containers: - name: dns-backend image: registry.k8s.io/example-dns-backend:v1 ports: - name: backend-port containerPort: 8000 </del>

master-blocking: gce-cos-master-alpha-featuresKubernetes e2e suite:Failed runs: 09-16 09:58 CDT 09-15 23:41 CDT Both of these runs were on the same commit which also has passing builds. The test fails due to a , specifically when attempting to connect to the DNS backend service . The error indicates that the DNS backend service is either not running or not accessible on the specified address and port. This caused multiple retries in the test, with the pod failing to establish a connection to the backend service each time. As a result, the expected output "Hello World!" was not found in the logs, leading to the failure of the test. The service being unreachable could be due to several potential issues: misconfigured DNS, networking issues within the cluster, or the DNS backend service failing to start properly.Further investigation into the DNS service deployment and networking might be required to resolve this issue. There were two other failures ( , ) on the same commit that seemed to be related to cluster instance not being found: This caused failure when attempting to SSH into the instance and retrieve logs (kube-, , etc.). For those, the cluster setup (e2e-) likely failed, as no nodes were found, resulting in no instances being available. /sig network /kind flake cc:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /kind flake
This test should not be running in this job, it was picked recently ! since we already have coverage I think is better to remove it, is also using images that the sources are not available anymore


Question: cluster/aws/templates/configure-vm-aws.sh:  kubelet_api_servers: '${KUBELET_APISERVER}' cluster/centos/util.sh:  local node_ip=${node#*@} cluster/gce/config-default.sh:# reloads <os_distro>/helper.sh in the gap between when the master is created <ins> cluster/gce/config-test.sh:# reloads <os_distro>/helper.sh in the gap between when the master is created </ins> cluster/gce/configure-vm.sh:  advertise_address: '${EXTERNAL_IP}' cluster/gce/configure-vm.sh:  api_servers: '${KUBERNETES_MASTER_NAME}' cluster/gce/configure-vm.sh:  cloud_config: ${CLOUD_CONFIG}

As of (specifically ), running against GCE without setting a node image defaults it to containervm, whereas before it defaulted it to the master image. For example, my config of: brings up a coreos master and then launches nodes with images. Before the above-referenced change this gave me my expected coreos nodes. Was this change intentional? The easiest fix that comes to mind for me is only doing the new containervm defaulting if is gci. cc
Yeah, this was something I was concerned too while making the change. However, the decision seems to have been that we decided to default to GCI, yet still default to the debian based containervm. Also, if we are to seriously start supporting hybrid clusters, i.e., master and nodes run on different OS distros, which seems true according to , this old assumption of defaults to needs to disappear anyways. cc/ Thoughts?
That sounds reasonable to me. We clearly didn't think about your use case whilst making this change.
I don't see the issue with defaulting to non-hybrid while supporting hybrid.. Defaulting seems totally reasonable even in a world where hybrid support is possible. A user could explicitly control that they want hybrid via setting them to different values regardless of the defaulting, or wherever hybrid is toggled on different defaulting could be done. I left a few questions on the referenced original issue which, if/when answered, might help me in making a PR to fix this for my usecase more cleanly.
Agreed. I am making a fix to this.