Question: return &DeviceInfo{deviceName, p.major, p.minor}, nil } <del> func (i *RealFsInfo) GetDirFsDevice(dir string) (*DeviceInfo, error) { buf := new(syscall.Stat_t) err := syscall.Stat(dir, buf) if err != nil { return nil, fmt.Errorf("stat failed on %s with error: %s", dir, err) } // The type Dev in Stat_t is 32bit on mips. major := major(uint64(buf.Dev)) // nolint: unconvert minor := minor(uint64(buf.Dev)) // nolint: unconvert for device, partition := range i.partitions { if partition.major == major && partition.minor == minor { return &DeviceInfo{device, major, minor}, nil } } </del> <ins> func (i *RealFsInfo) mountInfoFromDir(dir string) (*mount.MountInfo, bool) { </ins> mount, found := i.mounts[dir] // try the parent dir if not found until we reach the root dir // this is an issue on btrfs systems where the directory is not

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubelet v1.19.0 doesn't start on nodes with CRI-O 1.17.2 (storagedriver="btrfs") with BTRFS as root filesystem. Related/previous issues: This error shows up in the journal: The is referring to the btrfs root filesystem, mounted at the time. : : What you expected to happen: Kubelet starts and detects the filesystem backing , even if it is BTRFS. How to reproduce it (as minimally and precisely as possible): Install kubeadm/kubectl 1.19.0 Install CRI-O, I used 1.17.2 from: Configure CRI-O to use Perform "kubeadm init" creating a new cluster Observe journal (check for kubelet issues) Anything else we need to know?: A possible workaround is to make sure a bind mount exists which allows kubelet's logic to find the backing fileystem. Eg. add the following fstab entry and then perform : The resulting mount should look somewhat like this: The workaround has been tested and seems to make kubelet work as expected. Environment: Kubernetes version: 1.19.0 OS: Debian Bullseye (Testing) Kernel: Linux 5.8.5 Others: The package has been installed
/sig node
cc xref
I am encounter the same issue. thank you for your workaround. It seems to work.
I got the same issue with latest stable Docker on Debian (btrfs) and k8s v1.19.2.
I confirm the same issue shows up with on btrfs (on LVM on LUKS, Debian buster), running k3s. No btrfs subvols. The workaround works for me; for k3s I also need to bind mount . I did not experience this issue with 1.18.
This also reproduces with Docker runtime. Workaround with bind mount helps, however automatically unmounts it so don't get confused. is on root that is mounted like this: , and docker is configured to use .

Reproduced with k8s 1.19.3 on Gentoo. The rootfs is btrfs, is on rootfs, docker is configured with storage drigver . automatically mount and unmount it on . Also confirmed bind mount workaround helps.
/assign
I think almost anyone using BTRFS as the root filesystem has encountered this issue. If is merged and applied to the kubernetes, this can be fixed. Please take a look at this ?
The has been merged to google/cadvisor. I think if Kubernetes v1.21 includes the fix by using google/cadvisor v0.39, the issue will be resolved.
Just a note that I too have stumbled on this issue after reconfiguring CRI-O storage to use btrfs driver pointing with graphroot directly to a BTRFS subvolume (the root filesystem is not BTRFS). After reading this issue I tried mounting the BTRFS subvolume explicitly on some mountpoint and pointed with graphroot to this explicit subvolume mountpoint which solved the problem. The problem seems to occur only when pointing to a subvolume within some parent subvolume as this is not a real mountpoint.
Ok, so I just want to make sure I understand. Currently when i try to run kubeadm to create a new cluster while using btrfs, it gives me With Kubernetes v1.21, that will no longer be the case?
In my experience (with CRI-O 1.19 and k8s 1.19), the btrfs driver (in ) is supported, but the combination does not work correctly when graphroot is pointed to a btrfs subvolume within some parent subvolume (I don't have experience with graphroot poited to some subdirectory of a default subvolume and/or when default subvolume is also a root filesystem, just when pointed directly to a subvolume within some parent subvolume which is not a root filesystem). For example, when btrfs filesystem is mounted like the following (): when I then create a subvolume: then use it as a graphroot (): But when I explicitly mount the subvolume to some mountpoint (): then use the explicitly mounted path as the graphroot (): it works correctly without any problems.

I guess some logic (either in CRI-O/containers or in kubelet) tries to find the "mountpoint" of the filesystem used for graphroot or something related and can't do that correctly when graphroot is located in a btrfs subvolume that is not explicitly mounted somewhere.
Ah, I see what you meant by saying that is not supported in . I haven't checked that since I created k8s clusted with default (overlay) driver and later migrated to . It is possible that disallows using because of problems like the one described above. But in my experience, when a subvolume is mounted explicitly, the btrfs works as expected - at least I didn't have problems with it.
Can the fix be backported to 1.19? only supports updating one version at a time: () So this leaves people on btrfs with with no upgrade path to which is not really nice.
We've discussed this issue at Regarding Maybe, there are quite a lot of people using btrfs than I expected. What do you think about this? Thanks!
I am running up against this issue as well. I am stuck with using k8s on 1.18
It sounds like for full fix we'll need to cherrypick to be included in k8s 1.19 and 1.20. To do that the following is needed: to cAdvisor v0.37 and v0.38 branches k8s 1.19 uses cAdvisor v0.37.3 - k8s 1.20 uses cAdvisor v0.38.6 - new cAdvisor patch release for v0.37 and v0.38 against k8s on 1.19 and 1.20 bumping to newly released cAdvisor (v0.37 / v0.38) with the fix I'm happy to cut the new cAdvisor release from my side if you want to proceed and think this is critical enough bug that is worth cherrypicking.
New cAdvisor patch versions have been cut with the BTRFS fix: v0.37.4 - v0.38.7 - /cc


Question: if containerChanges.StartInfraContainer && (len(containerChanges.ContainersToStart) > 0) { glog.V(4).Infof("Creating pod infra container for %q", podFullName) podInfraContainerID, err = dm.createPodInfraContainer(pod) <ins> if err != nil { glog.Errorf("Failed to create pod infra container: %v; Skipping pod %q", err, podFullName) return err } </ins> // Call the networking plugin <del> if err == nil { err = dm.networkPlugin.SetUpPod(pod.Namespace, pod.Name, podInfraContainerID) } </del> <ins> err = dm.networkPlugin.SetUpPod(pod.Namespace, pod.Name, podInfraContainerID) </ins> if err != nil { glog.Errorf("Failed to create pod infra container: %v; Skipping pod %q", err, podFullName) <ins> // Delete infra container if delErr := dm.KillContainerInPod(kubecontainer.ContainerID{ ID:   string(podInfraContainerID), Type: "docker"}, nil, pod); delErr != nil { glog.Warningf("Clear infra container failed for pod %q: %v", podFullName, delErr) } </ins> return err }

I'm seeing a bug where the network plugin fails to configure networking, and yet the containers for this pod are still deployed (without networking configured). It looks like the following is occurring: 1) Kubelet attempts to create infra container in 2) Network plugin fails, returns error code. 3) Kubelet detects error and skips pod creation but does not tear down the infra container 4) On the next call, the kubelet detects that the infra container is running and deploys the other containers in the pod. The fix here is for the kubelet to clean up the infra container in when it receives the error from the network plugin and before it returns. Here are some logs illustrating the steps I described above:
Cleaning up infra container on network plugin error sounds reasonable/


Question: return nil } <del> // Serialize the current configuration of the object from the server. current, err := runtime.Encode(encoder, info.Object) if err != nil { return cmdutil.AddSourceToErr(fmt.Sprintf("serializing current configuration from:n%vnfor:", info), info.Source, err) } // Retrieve the original configuration of the object from the annotation. original, err := kubectl.GetOriginalConfiguration(info) if err != nil { return cmdutil.AddSourceToErr(fmt.Sprintf("retrieving original configuration from:n%vnfor:", info), info.Source, err) } // Create the versioned struct from the original from the server for // strategic patch. // TODO: Move all structs in apply to use raw data. Can be done once // builder has a RawResult method which delivers raw data instead of // internal objects. versionedObject, _, err := decoder.Decode(current, nil, nil) if err != nil { return cmdutil.AddSourceToErr(fmt.Sprintf("converting encoded server-side object back to versioned struct:n%vnfor:", info), info.Source, err) } // Compute a three way strategic merge patch to send to server. patch, err := strategicpatch.CreateThreeWayMergePatch(original, modified, current, versionedObject, true) if err != nil { format := "creating patch with:noriginal:n%snmodified:n%sncurrent:n%snfrom:n%vnfor:" return cmdutil.AddSourceToErr(fmt.Sprintf(format, original, modified, current, info), info.Source, err) } </del> helper := resource.NewHelper(info.Client, info.Mapping) <del> _, err = helper.Patch(info.Namespace, info.Name, api.StrategicMergePatchType, patch) </del> <ins> patcher := NewPatcher(encoder, decoder, info.Mapping, helper) patchBytes, err := patcher.patch(info.Object, modified, info.Source, info.Namespace, info.Name) </ins> if err != nil { <del> return cmdutil.AddSourceToErr(fmt.Sprintf("applying patch:n%snto:n%vnfor:", patch, info), info.Source, err) </del> <ins> return cmdutil.AddSourceToErr(fmt.Sprintf("applying patch:n%snto:n%vnfor:", patchBytes, info), info.Source, err) </ins> } if cmdutil.ShouldRecord(cmd, info) { <del> patch, err = cmdutil.ChangeResourcePatch(info, f.Command()) </del> <ins> patch, err := cmdutil.ChangeResourcePatch(info, f.Command()) </ins> if err != nil { return err }

From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct.

Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now?

I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016


Question: package get import ( <ins> "encoding/json" </ins> "reflect" "strings" "testing"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: seems to have regressed in v1.13.0 and no longer sorts by creation timestamp. What you expected to happen: should sort by creation timestamp. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: N/A Environment: Kubernetes version (use ): <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig cli
/assign
bisected to () /assign
opened to fix
/assign


Question: local i for i in $(seq 1 "${times}"); do local out <del> # shellcheck disable=SC2086 # Disabling because "${extra_args}" needs to allow for expansion here if out=$(curl --max-time "${maxtime}" -gkfs $extra_args "${url}" 2>/dev/null); then </del> <ins> if out=$(curl --max-time "${maxtime}" -gkfs "${@:6}" "${url}" 2>/dev/null); then </ins> kube::log::status "On try ${i}, ${prefix}: ${out}" return 0 fi

What happened: After running ./hack/update-openapi- with the latest code, api/openapi- file became empty even when no changes were made in the code. What you expected to happen: After running ./hack/update-openapi- with the latest code, api/openapi- file should not be changed when no changes were made in the code. When I tried the same thing a few week back, I didn't encounter this problem. How to reproduce it (as minimally and precisely as possible): Download the latest kubernetes/kubernetes code and run the following script to update openapi spec. api/openapi- became empty after this. Anything else we need to know?: There's a line in update-openapi- that runs a curl command . This line was changed a few days ago by this commit: PR: I checked out the commit before and ran /update-openapi- It works. is not empty any more. Is there any new setting required before running ./hack/update-openapi- now? Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
cc
Hi, thanks for the report, but it is a bit weird that I could not reproduce locally: and besides, completes successfully on my machine. Could you reproduce?
Someone else also ran into this same problem and asked a question on sig-api-machinery slack yesterday. So I'm not the only one who ran into this.
could you change the args passed to curl in to and rerun the script to see if there is any error?
since it is already late here(UTC+8), I would check this issue again tomorrow
I was the "someone else" who ran into this problem :) I changed the script to use The result: Actually, it was 403 without the "S", too, but that is visible only if curl is executed in verbose mode.
Hi, thanks for your information. Given that the apiserver returned 403, I suspect the flag passed to curl has no effect in your environment, would you mind posting the output of ? I am using curl 7.64.1: In addition, I think you could try changing the curl command in to the following: and see if the problem is solved.

I tried the following and it worked for me. is no longer empty. curl version:
Thanks for your confirmation, and I have filed to address this issue.
I can confirm, it works with the changed curl command.
/triage accepted


Question: fakeClock.Step(2 * notifierRefreshInterval) manager.synchronize(diskInfoProvider, activePodsFunc) } <ins> func TestManagerWithLocalStorageCapacityIsolationOpen(t *testing.T) { podMaker := makePodWithLocalStorageCapacityIsolationOpen summaryStatsMaker := makeDiskStats podsToMake := []podToMake{ {name: "empty-dir", requests: newResourceList("", "900Mi", ""), limits: newResourceList("", "1Gi", "")}, {name: "container-ephemeral-storage-limit", requests: newResourceList("", "", "900Mi"), limits: newResourceList("", "", "800Mi")}, {name: "pod-ephemeral-storage-limit", requests: newResourceList("", "", "1Gi"), limits: newResourceList("", "", "800Mi")}, } pods := []*v1.Pod{} podStats := map[*v1.Pod]statsapi.PodStats{} for _, podToMake := range podsToMake { pod, podStat := podMaker(podToMake.name, podToMake.priority, podToMake.requests, podToMake.limits, podToMake.memoryWorkingSet) pods = append(pods, pod) podStats[pod] = podStat } summaryProvider := &fakeSummaryProvider{result: summaryStatsMaker("1Gi", "200Mi", podStats)} config := Config{ MaxPodGracePeriodSeconds: 5, PressureTransitionPeriod: time.Minute * 5, Thresholds: []evictionapi.Threshold{ { Signal:   evictionapi.SignalAllocatableMemoryAvailable, Operator: evictionapi.OpLessThan, Value: evictionapi.ThresholdValue{ Quantity: quantityMustParse("1Gi"), }, }, }, } podKiller := &mockPodKiller{} diskGC := &mockDiskGC{err: nil} nodeRef := &v1.ObjectReference{Kind: "Node", Name: "test", UID: types.UID("test"), Namespace: ""} fakeClock := testingclock.NewFakeClock(time.Now()) diskInfoProvider := &mockDiskInfoProvider{dedicatedImageFs: false} mgr := &managerImpl{ clock:                         fakeClock, killPodFunc:                   podKiller.killPodNow, imageGC:                       diskGC, containerGC:                   diskGC, config:                        config, recorder:                      &record.FakeRecorder{}, summaryProvider:               summaryProvider, nodeRef:                       nodeRef, localStorageCapacityIsolation: true, dedicatedImageFs:              &diskInfoProvider.dedicatedImageFs, } activePodsFunc := func() []*v1.Pod { return pods } evictedPods := mgr.synchronize(diskInfoProvider, activePodsFunc) if podKiller.pod == nil { t.Fatalf("Manager should have selected a pod for eviction") } if diff := cmp.Diff(pods, evictedPods); diff != "" { t.Fatalf("Unexpected evicted pod (-want,+got):n%s", diff) } } </ins>

As I was reading the test code, I noticed that we are missing a pretty large chunk of logic around localStorageCapacityIsolation. This feature was GA in 1.25 so it would be ideal to have unit test coverage around this code. This code monitors whether or not there is a violation in temporary storage and if there is it would evict the pod. One should be able to intiailize the manager with of true and then write some test cases to verify this behavior. /help /sig node /good-first-issue /kind bug
/cc do you have any recollection why there weren't tests for this path?
I can help with this. can you assign me after jingxu replies to you?(for more context here)
that would be great !
you're welcome. do you have any context here that why didn't cover localStorageCapacityIsolation?
I don't recall a specific reason. The feature was implemented a while back.
/assign
Hey Are you still working. If not can I take this up?
hi, yes I'm still working on it
hii I am new to k8s , so i wanna know can I use windows os for it? if yes would you show me how to get nodes...
hi, the answer is yes you can use windows and I think if you need support you better open another issue and @ me there, as this issue is not for discuss how to run k8s on windows
/remove-kind bug /kind feature /triage accepted
I see and I am wondering if we have duplicated efforts here. did you end up testing this path in our above PR?
Hey, is someone working on this issue ? or let me know if someone need help
Happy to help with this. Do you still need help
hey, is this currently still unassigned
has a ready for approval. /remove-good-first-issue /remove-help


Question: metav1.AddToGroupVersion(scheme, metav1.SchemeGroupVersion) utilruntime.Must(example.AddToScheme(scheme)) utilruntime.Must(examplev1.AddToScheme(scheme)) <ins> utilruntime.Must(example2v1.AddToScheme(scheme)) </ins> } func newTestCacher(s storage.Interface) (*Cacher, storage.Versioner, error) {

The test is flaking: /assign /sig api-machinery /priority important-soon
Ok - it's fairly easy to reproduce by just adding into the test I will validate the fix.
/triage accepted


Question: return num } <del> func countEndpointsSlicesNum(epList *discoveryv1.EndpointSliceList) int { </del> <ins> // isIPv6Endpoint returns true if the Endpoint uses IPv6 addresses func isIPv6Endpoint(e *v1.Endpoints) bool { for _, sub := range e.Subsets { for _, addr := range sub.Addresses { if len(addr.IP) == 0 { continue } // Endpoints are single family, so it is enough to check only one address return netutils.IsIPv6String(sub.Addresses[0].IP) } } // default to IPv4 an Endpoint without IP addresses return false } func countEndpointsSlicesNum(epList *discoveryv1.EndpointSliceList, addressType discoveryv1.AddressType) int { </ins> // EndpointSlices can contain the same address on multiple Slices addresses := sets.Set[string]{} for _, epSlice := range epList.Items { <ins> if epSlice.AddressType != addressType { continue } </ins> for _, ep := range epSlice.Endpoints { if len(ep.Addresses) > 0 { addresses.Insert(ep.Addresses[0])

All the dual stack jobs ! ,%20ipvs,%20dual,%20master All dual stacks tests Since I modified this helper function to fix a flake ,%20ipvs,%20dual,%20master The function must consider only one IP family to verify the number of endpoints /assign /sig network /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return true } <ins> func TestCacheIncreaseDoesNotBreakWatch(t *testing.T) { store := newTestWatchCache(2, &cache.Indexers{}) now := store.clock.Now() addEvent := func(key string, rv uint64, t time.Time) { event := &watchCacheEvent{ Key:             key, ResourceVersion: rv, RecordTime:      t, } store.updateCache(event) } // Initial LIST comes from the moment of RV=10. store.Replace(nil, "10") addEvent("key1", 20, now) // Force "key1" to rotate our of cache. later := now.Add(2 * eventFreshDuration) addEvent("key2", 30, later) addEvent("key3", 40, later) // Force cache resize. addEvent("key4", 50, later.Add(time.Second)) _, err := store.GetAllEventsSince(15) if err == nil || !strings.Contains(err.Error(), "too old resource version") { t.Errorf("unexpected error: %v", err) } } </ins> func BenchmarkWatchCache_updateCache(b *testing.B) { store := newTestWatchCache(defaultUpperBoundCapacity, &cache.Indexers{}) store.cache = store.cache[:0]

Opening as an follow up analysis from Merging unexpectedly resulted in a huge flakiness of at least one of the apimachinery tests: That is definitely unexpected, because that PR only changes the size of watchcache, which no e2e tests should rely on. So I started digging deeper into it and indeed that PR also uncovered the bug that was introduced earlier. Here is what is happening: made the watch cache size dynamic however, there was a place in watch cache that relied on the following invariant: "when watchache becomes full, it is full until the next relist" that place is this one: with the change that made watchcache size dynamic, it may no longer be true, i.e.: (a) watch cache was full (b) we are increasing its size (c) it is no longer full, but we no longer can't serve (at least not always) everything since listResourceVersion This is exactly what was happening in the failed tests: the service was updated well back in the past the watchcache initially rotated couple times (no need to increase it) [which pulled our change out of it] the its size was increased that resulted in cache becoming not full so the incoming watch should have returned "too old", but was accepting it (due to the listResourceVersion 0 condition being true), but in fact wasn't able to serve it Bonus point: in "kind" tests we were observing tests taking 30+ minutes - this seem to be a bug in kube-apiserver configuration, because the watch in the test doesn't set TimeoutSeconds, and: (a) in GCE it is default in kube-apiserver to sth random from [5m, 30m]: (b) in kind apparently it isn't... /kind bug /sig apimachinery /assign - FYI
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Opening as an follow up analysis from That is definitely unexpected, because that PR only changes the size of watchcache, which no e2e tests should rely on.

made the watch cache size dynamic however, there was a place in watch cache that relied on the following invariant: the service was updated well back in the past the watchcache initially rotated couple times (no need to increase it) [which pulled our change out of it] the its size was increased that resulted in cache becoming not full so the incoming watch should have returned "too old", but was accepting it (due to the listResourceVersion 0 condition being true), but in fact wasn't able to serve it Bonus point: in "kind" tests we were observing tests taking 30+ minutes - this seem to be a bug in kube-apiserver configuration, because the watch in the test doesn't set TimeoutSeconds, and: (a) in GCE it is default in kube-apiserver to sth random from [5m, 30m]: (b) in kind apparently it isn't... /kind bug /sig apimachinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Working on the fix right now
for now contains a test reproducing the issue Fix coming soon.
is ready for review
it seems TimeoutSeconds is configured explicitely on GCE tests to 300 sec KIND is using Kubeadm under the hood and I could not find any place where those flags are configured, but then I think it should be taken the default values for TimeoutSeconds, no? EDIT Actually, it seems that it does take the default, and min-request-timeout takes precedence, that means it takes 1800s it seems we should set this apiserver option to a lower value, should we do it in KIND itself?
I would want to speak to the kubeadm team first, we avoid deviating from their defaults as much as possible so we can test them.

cc
i see the tests under "client-go should negotiate" are not conformance yet, which makes them less likely to be run outside of k8s testing. i personally don't think our tools used for e2e (GCE/kube-up included) should be configuring these kube-apiserver flags. the tests should behave well with any deployer-X that relies on the apiserver sane defaults.


Question: default: errString := "you must specify two arguments: verb resource or verb resource/resourceName." usageString := "See 'kubectl auth can-i -h' for help and examples." <del> return errors.New(fmt.Sprintf("%sn%s", errString, usageString)) </del> <ins> return fmt.Errorf("%sn%s", errString, usageString) </ins> } }

What happened: If you use to test whether a user can approve certificate signing requests, you get a response indicating that is not a valid verb An example of the verb being used can be seen What you expected to happen: should be recognized as a valid verb on objects. How to reproduce it (as minimally and precisely as possible): Run Anything else we need to know?: I think that needs to be to the kubectl auth can-i code Environment: Kubernetes client and server versions (use ): Cloud provider or hardware configuration: OS (e.g: ):
This issue is currently awaiting triage. SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think is a valid verb for kubectl auth command. Valid verbs are in
so in that line of code there are a number of verbs which are only usuable on specific objects for example which works on users. AFAIK the point of that line is to inform kubectl what the valid verbs are on any object in Kubernetes, so that it knows what's possible when users use the command. the verb is valid on certificate objects as shown in the docs example I listed. Without being present in that line of code, will incorrectly state that there is no verb in Kubernetes.
I tested it, seems working when we add in that list. But the question is should we add it or not?
There have been cases where Kubectl has had additions to handle non-standard resources/verbs in the past (e.g. I raised a couple of years back). AFAIK the explicit list of verbs here is designed to address this point, it's just not got approve in it. You could ask SIG-Auth about it, I'm sure they'd have more info.
/transfer kubenertes /sig auth Decision can be made by sig-auth
Something went wrong or the destination repo kubernetes/kubenertes does not exist. <detailsIn response to : /transfer kubenertes /sig auth Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes/kubernetes
Something went wrong or the destination repo kubernetes/kubernetes/kubernetes does not exist. <detailsIn response to : /transfer kubernetes/kubernetes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes
/triage accepted
This will work even without adding verb there, adding verb just removes the warning. Submitted a PR to add that. /assign


Question: return } proxier.network = *updatedNetwork <del> var providerAddress string for _, rs := range proxier.network.remoteSubnets { _, ipNet, err := net.ParseCIDR(rs.destinationPrefix) if err != nil { klog.Fatalf("%v", err) } if ipNet.Contains(net.ParseIP(ep.ip)) { providerAddress = rs.providerAddress } if ep.ip == rs.providerAddress { providerAddress = rs.providerAddress containsNodeIP = true } } </del> <ins> providerAddress := proxier.network.findRemoteSubnetProviderAddress(ep.ip) </ins> if len(providerAddress) == 0 { klog.Infof("Could not find provider address for %s. Assuming it is a public IP", ep.ip) providerAddress = proxier.nodeIP.String() <del> containsPublicIP = true </del> } hnsEndpoint := &endpointsInfo{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In Overlay mode, When 2 service has the same backend node IP, the LB was not programmed correctly causing datapath issues. What you expected to happen: Service data path to work properly How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /sig windows These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: } // Pop removes the head of the active queue and returns it. It blocks if the <del> // activeQ is empty and waits until a new item is added to the queue. It also // clears receivedMoveRequest to mark the beginning of a new scheduling cycle. </del> <ins> // activeQ is empty and waits until a new item is added to the queue. </ins> func (p *PriorityQueue) Pop() (*v1.Pod, error) { p.lock.Lock() defer p.lock.Unlock()

What happened: Kubernetes scheduler may leave a single pod stuck in pending state after a worker node is reloaded/reinstalled. This problem appears to have been introduced in Kubernetes v1.12.4 and v1.13.1. We have not been able to recreate the problem in our testing on earlier releases. The key to the failure is that the worker being reloaded/reinstalled must be the only scheduling option available. The pending pod is eventually scheduled if a new scheduler leader is elected or the node status changes. This appears to be a timing bug since the problem does not always occur. What you expected to happen: No pods are stuck in pending state. How to reproduce it (as minimally and precisely as possible): Start with a single node cluster running a handful of pods. Cordon, drain and then reload/reinstall the node. Since this is a timing bug, it may take several iterations to recreate the problem. Environment: Kubernetes version (use ): 1.12.4, 1.13.1 and 1.13.2 Cloud provider or hardware configuration: VMs running in VirtualBox OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Custom ansible scripts Others: N/A
/sig scheduling
Events from the pending pod: Events from the node:
does "reload/reinstall" mean info is still kept in etcd? or (or equivalent steps) is executed to remove that node entirely? And I assume the "reloaded" node keeps the same spec and IP/node name with previous one?
the info is still kept in etcd. And the node keeps the same spec, IP and name as it had before. The reload does the following: and drain the node the VM the VM again with the same spec the node
What's the taint/toleration accordingly?
there were no taints to the node during this process. The pods are part of a deployment with only basic tolerations provided by Kubernetes. Deleting the pod will get it scheduled even though the taints and tolerations are unchanged. Simply restarting the scheduler leader will get the pod scheduled as well.
Can you share and ?
The symptom (esp. "a single pod stuck in pending state") is quite similar as issue .

I think why this occurs since v1.12.4 is related the PR - which was delivered to master branch, and backported to v1.12.4, and it appears to reveal higher chance of racing. cc/ Basically we have 2 solutions to fix this: the real root cause of the racing condition and give fix (I haven't got luck to reproduce this yet) that we have a solution () to alleviate this kind of problem, could you try to run your test on a) master branch or b) v1.12.4 with cherrypicked to see if the issue is resolved? and does this fits your business need? (in other words, the single pod may be pending for 30 seconds at most)
I'll recreate and provide you the details. However, the pod that is stuck pending is different every time.
Thanks, I'll work on the recreate and test of the cherrypick. That PR certainly seems promising.
Node YAML after recreate:
Pod YAML after recreate:
the cherrypick doesn't look trivial for 1.12 or 1.13. Give me some time to setup an environment to test on 1.14. Unless you have another way?
Yeap, it's not a trivial task to rebuild the source code. Let me know if you need any help. I suppose you're going to try on latest 1.14 tag? e.g. v1.14.0-alpha.1? Right now I'm working on the 1st solution I mentioned in The challenge is I can't reproduce this in my local dev env (I'm running on source code branch of v1.12.4) yet, so my current plan is to retry reproducing your issue to make my solution more convincing. If still no luck, I will send out the PR for reviewing early next week.
And let me confirm with your statement: When the issue occurs, everytime there is only one single pod stuck in pending state, right?
Still don't have luck in reproducing the issue. But I've prepared a PR and put detailed analysis there. Any comments are welcome.
No, the problem does not occur every time. It can take as many as 7 retries before reproducing it.

Sry, I should have made it more clear - I wanted to confirm that among all the times the issue is reproducible, is that everytime there is only one single pod in pending state?
correct, every time that I've reproduced the problem there has been only one pod in pending state.
Thanks. To some extent, it's consistent with the deduction in my PR.
is it possible to have the necessary fixes cherry-picked to at least 1.12 and 1.13? That would make it easier for me to build and test?
Yeap, cherrypick PRs ( onto 1.13, onto 1.12) were created and will be reviewed/approved soon.
Thank you for the cherrypick PRs. I rebuilt 1.13 with the cherrypick PR and still saw the problem on 2 of 10 recreate attempts. Without the fix, I saw a similar failure rate.
the fix is only ready in master right now. Did you rebuild 1.13 by manually running “git cherrypick <this commit id” on release-1.13? Or directly built from latest src code of release-1.13?
I manually rebuilt 1.13.2 with
/reopen I will take another look.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I've verified is able to fix your issue thoroughly. And cherrypick PRs have been raised (1.12 , 1.13 ).
Thank you.
Thanks, and K8s 13.3 already has another change () that should alleviate the problem significantly. K8s 13.4 will have the definitive fix. Given that the root-cause is fixed, we can close this issue.


Question: ------ TODO: Document events (refer to another doc for details) <ins> API Documentation ----------------- API documentation can be found at [http://kubernetes.io/third_party/swagger-ui/](http://kubernetes.io/third_party/swagger-ui/). </ins>

The schema understood by the Kubernetes configuration file is not described any where. The current practice mostly is copy/paste existing samples (for example ) and edit them. There is a need to define a comprehensive list of elements, and their documentation, so that other examples can leverage them.
Configuration files specify exactly our API objects. See:
This is exactly what I was looking for but not easily googlable, or documented clearly.
Could you please put a link to the swagger UI here?
The only mention in the public docs is here: which links directly to the struct definition on github. It appears that the Swagger docs have been deprecated, and I'm unable to find any documentation for the KubeletConfiguration resource kind in the new docs. It's referred to in at least one place, but is still not actually covered:


Question: glog.Errorf("computeHostTags exceeded maxPages=%d for Instances.List: truncating.", maxPages) } } if len(tags) == 0 { <del> glog.V(2).Info("No instances had tags, creating rule without target tags") </del> <ins> return nil, fmt.Errorf("No instances found") </ins> } return tags.List(), nil }

In some custom setup cases the prefix-matching logic for GCE load-balancers can break and end up creating a firewall rule with no tags. We should instead refuse to create the rule. In fact, if we find it, we should probably delete the rule, but that's maybe too aggressive. Looking at the logic, if we return error in the tag-selection, we'll abort the whole load-balancer creation, which is unfortunate, but not TERRIBLE. We could just silently not create the firewall but not return error, which will just retry over and over.
/cc
Are you active on this? may have some starter-project bandwidth to help.
Not active. It would indeed be a good starter project :)


Question: }, expectSpecifiedNodePorts: []int{}, }, <ins> { name: "Add new ServicePort with a different protocol without changing port numbers", oldService: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Spec: api.ServiceSpec{ Selector:        map[string]string{"bar": "baz"}, SessionAffinity: api.ServiceAffinityNone, Type:            api.ServiceTypeNodePort, Ports: []api.ServicePort{ { Name:       "port-tcp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolTCP, NodePort:   30053, }, }, }, }, newService: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Spec: api.ServiceSpec{ Selector:        map[string]string{"bar": "baz"}, SessionAffinity: api.ServiceAffinityNone, Type:            api.ServiceTypeNodePort, Ports: []api.ServicePort{ { Name:       "port-tcp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolTCP, NodePort:   30053, }, { Name:       "port-udp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolUDP, NodePort:   30053, }, }, }, }, expectSpecifiedNodePorts: []int{30053, 30053}, }, { name: "Change service type from ClusterIP to NodePort with same NodePort number but different protocols", oldService: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Spec: api.ServiceSpec{ Selector:        map[string]string{"bar": "baz"}, SessionAffinity: api.ServiceAffinityNone, Type:            api.ServiceTypeClusterIP, Ports: []api.ServicePort{{ Port:       53, Protocol:   api.ProtocolTCP, TargetPort: intstr.FromInt(6502), }}, }, }, newService: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Spec: api.ServiceSpec{ Selector:        map[string]string{"bar": "baz"}, SessionAffinity: api.ServiceAffinityNone, Type:            api.ServiceTypeNodePort, Ports: []api.ServicePort{ { Name:       "port-tcp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolTCP, NodePort:   30053, }, { Name:       "port-udp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolUDP, NodePort:   30053, }, }, }, }, expectSpecifiedNodePorts: []int{30053, 30053}, }, </ins> } for _, test := range testCases {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT /kind bug What happened: I tried to submit a Service Replace () to the kube-apiserver and this results the kube-apiserver container crashing and restarting. PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenameWhat you expected to happen: I expected the server not to crash and to get validation errors if the request was invalid. How to reproduce it (as minimally and precisely as possible): Create a service, get the resourceVersion and clusterIP from the created resource GET localhost:8001/api/v1/namespaces/<namespace/services/<servicenameSubmit a replace command to PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenamewith body like If the same request is submitted with the nodePort fields in the ports array then the request succeeds. I get that I may be sending an invalid request without a node port (though shouldn't the merge deal with that?) but it shouldn't take down the API server. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T23:29:08Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Bare metal cluster OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: Others:
thanks for reporting this! Seems like this is similar to and should be fixed by . can you PTAL that PR to see if it's good to merge?
/sig api-machinery
Duplicate of

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Re-applying a NodePort service that has two ports fails and crashes the API server: What you expected to happen: Do not crash, service is configured correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: on VBox provider in macOS OS (e.g. from /etc/os-release): macOS/Minikube Kernel (e.g. ): Install tools: Others:
Seems that in func updateNodePorts, it ignored protocol when Detecting duplicate nodeport, I would have a fix on it :)
/sig network
FYI: I think this would be fixed in
/cc Maybe this is the time to have another review on and merge it :)
/assign
Fine, That makes sense

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /cc /assign What happened: a ClusterIP type service the service. Change it's type to NodePort and add a Port. Note: the new port has same port number but protocol with the existing Port. the updated service. The TCP Port was eaten. What you expected to happen: The updated service has both TCP and UDP ports. How to reproduce it (as minimally and precisely as possible): See the step above. Anything else we need to know?: If I create a NodePort service with the above two ports, it works. I write a test case to verify this case. I got a panic. The root cause is as below. We should take another look at this logic. BTW: changed some related code but it doesn't change the logic here. So I believe this bug is not introduced by . I also tested this on v1.5.1, it has the same issue. Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I will be OOO the next week. Since this bug is not so urgent, I will fix it ASAP when I'm back. Folks who are interested in this can take it :) BTW, I can join in the discussion from my phone.
I'm not sure this is a bug. In the first configuration yaml, you're defining the ports array twice. I'm the second, you're only defining it once. The ports array should only be defined once.
Did you mean the below yaml by ? Please note that the two ports have same port number but different protocols. This is allowed.
Yes, that's the one I meant. But note you have "ports:" twice.
Ah, thanks for pointing it out! This might be the cause. But I remembered I also tested the case without the second line, it had the same issue. I can't verify this now. Will test it when I'm back to my keyboard.
I try the case without the second line, the issue also exists.

So I believe there is a bug here. Seems like there might be two issues here: a port is lost as is described in this issue. we shouldn't allow to configure two lines in the manifest. i.e. we should add validation for this case.
/remove-priority awaiting-more-evidence
can you please provide (or edit) example YAMLs then to allow for attempts at duplication?
Update: reproducing without the extra line. a ClusterIP type service the service (). Change it's type to NodePort and add a Port which has the same port number but different protocol with the existing Port. the updated service. We can see that the UDP Port was not created. I have dug into the code and will send a patch to fix it.
PR sent. Reviews welcome! Thanks!
There are/were bugs in the patch implementation wrt ports, because they need compound merge keys.
Yeah I had some discussios with in . But we also do have a bug in our service updating codes. See:


Question: name: "no scale down stabilization, reuse recommendation element", key:  "", recommendations: []timestampedRecommendation{ <del> {10, time.Now().Add(-10 * time.Minute)}, {9, time.Now().Add(-9 * time.Minute)}}, </del> <ins> {10, now.Add(-10 * time.Minute)}, {9, now.Add(-9 * time.Minute)}}, </ins> currentReplicas:              100, // to apply scaleDown delay we should have current > desired prenormalizedDesiredReplicas: 3, expectedStabilizedReplicas:   3, expectedRecommendations: []timestampedRecommendation{ <del> {10, time.Now()}, {3, time.Now()}, </del> <ins> {10, now}, {3, now}, </ins> }, }, { name: "no scale up stabilization, reuse recommendation element", key:  "", recommendations: []timestampedRecommendation{ <del> {10, time.Now().Add(-10 * time.Minute)}, {9, time.Now().Add(-9 * time.Minute)}}, </del> <ins> {10, now.Add(-10 * time.Minute)}, {9, now.Add(-9 * time.Minute)}}, </ins> currentReplicas:              1, prenormalizedDesiredReplicas: 100, expectedStabilizedReplicas:   100, expectedRecommendations: []timestampedRecommendation{ <del> {10, time.Now()}, {100, time.Now()}, </del> <ins> {10, now}, {100, now}, </ins> }, }, { name: "scale down stabilization, reuse one of obsolete recommendation element", key:  "", recommendations: []timestampedRecommendation{ <del> {10, time.Now().Add(-10 * time.Minute)}, {4, time.Now().Add(-1 * time.Minute)}, {5, time.Now().Add(-2 * time.Minute)}, {9, time.Now().Add(-9 * time.Minute)}}, </del> <ins> {10, now.Add(-10 * time.Minute)}, {4, now.Add(-1 * time.Minute)}, {5, now.Add(-2 * time.Minute)}, {9, now.Add(-9 * time.Minute)}}, </ins> currentReplicas:              100, prenormalizedDesiredReplicas: 3, expectedStabilizedReplicas:   5, expectedRecommendations: []timestampedRecommendation{ <del> {10, time.Now()}, {4, time.Now()}, {5, time.Now()}, {3, time.Now()}, </del> <ins> {10, now}, {4, now}, {5, now}, {3, now}, </ins> }, scaleDownStabilizationWindowSeconds: 3 * 60, },

When configuring new behaviors section in HPA I two scenarios that I think are incorrect. First one: stabilizationWindowSeconds set up to 5 minutes for both scale up and scale down min replica count 2 Application consumes big amount of resources during warm-up 0 minute: current replica count: 2 1 minute: desired replica count: 3 - no scale up stabilization window worked 2 minute: desired replica count: 1 (!) - scale up to 3 (max number from previous recommendations during stabilization window period) ! Second one Stabilization window for scale up 0 Stabilization window for scale down 5 minutes min replica count 2 behavior set to scale up new pod in 2 minutes scaleUp: policies: type: Pods value: 1 periodSeconds: 120 What happens: 0 minute: 2 replicas 1 minute : performance spike scale up to 3 (desired value 4) 2 minute : no spike no scale up 3 minute: no spike scale up to 4(!) - supposedly desired value 1 and supposedly worked this way because of scale down window set to 5 minutes picking max value from that 5 minutes window that was 4 replicas Expected behaviors 1 Scenario no scale up for 5 minutes 2 Scenario correct measurement and no scale up when there is no need for it Environment: Kubernetes version (use ): client 1.19.3 Server 1.19.0 Cloud provider or hardware configuration: Azure aks
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
/assign josephburnett
The HPA controller keeps a flat history of recommendations for stabilization. ! When downscale stabilizing the maximum recommendation is selected. Purple is pre-stabilized recommendation. Green is final recommendation. Blue is max recommendation within window. ! When upscale stabilization the minimum recommendation is selected. ! However when both up and down scale stabilization is configured, the interpretation of the history changes depending on the direction of movement.

Red is the undesired, actuated recommendation. ! What we want is to keep the stabilized recommendation within the envelope of the minimum and maximum over configured stabilization windows. We should only move when the envelope forces a move. ! We first stabilize up and down, and check if the current value fits within it. If it does (or the range is "inverted"), we do nothing. If it does not, we emit the "nearest possible" value, which will be lower or upper bound, depending on where the current size is in relation to the range. (thanks and for the paragraph above)


Question: // if this is a pod in an older version AND there is a new version of this pod, record when // we started seeing this, otherwise delete the record (perhaps the node was drained) if nodesToVersions[pod.Spec.NodeName][newVersion] > 0 { <del> if _, ok := ageOfOldPod[string(pod.UID)]; !ok { ageOfOldPod[string(pod.UID)] = now </del> <ins> if _, ok := nodeToAgeOfOldPod[pod.Spec.NodeName][string(pod.UID)]; !ok { if _, ok := nodeToAgeOfOldPod[pod.Spec.NodeName]; !ok { nodeToAgeOfOldPod[pod.Spec.NodeName] = make(map[string]time.Time) } nodeToAgeOfOldPod[pod.Spec.NodeName][string(pod.UID)] = now </ins> } } else { <del> delete(ageOfOldPod, string(pod.UID)) </del> <ins> delete(nodeToAgeOfOldPod, pod.Spec.NodeName) </ins> } } // purge the old pods list of any deleted pods <del> for uid := range ageOfOldPod { if !podUIDs.Has(uid) { delete(ageOfOldPod, uid) </del> <ins> for node, uidToTime := range nodeToAgeOfOldPod { for uid := range uidToTime { if !podUIDs.Has(uid) { delete(uidToTime, uid) } } if len(uidToTime) == 0 { delete(nodeToAgeOfOldPod, node) </ins> } } deliberatelyDeletedPods = deliberatelyDeletedPods.Intersection(deletedPodUIDs)

/kind flake /sig apps This is an old flaking since , but the flaky rate is not that high. The rate in recent runs in is about 3/23 ~ 13%.
/priority backlog as the failure rate now is not high /cc old threads in and
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority backlog as the failure rate now is not high /cc old threads in and Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority backlog
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten still flaking
/triage accepted /assign


Question: Key:    key, Values: exp.Values, } <del> } else if exp.Key == key { newExp = exp </del> } else { <del> return nil, fmt.Errorf("unknown topology key: %v", exp.Key) </del> <ins> // Other topologies are passed through unchanged. newExp = exp </ins> } newTerm.MatchLabelExpressions = append(newTerm.MatchLabelExpressions, newExp) }

Unknown allowedTopology labels cause csi translation to fail: However, on GKE other labels are sometimes used in PV node affinity, such as or It's not common (100s of clusters as of July 2021), but seems enough that it should be dealt with. The suggestion is to just pass unknown topologies through. If they are used in the storageclass labels they will cause failures, but that is the current gce-pd behavior anyway. This reasoning probably holds for other clouds as well, although I haven't tested it myself. /assign /sig storage
/triage accepted


Question: break } if !found { <del> glog.Fatalf("Unable to find suitible network address in list: %v", addrs) </del> <ins> glog.Errorf("Unable to find suitible network address in list: '%v'n"+ "Will try again in 5 seconds. Set the public address directly to avoid this wait.", addrs) time.Sleep(5 * time.Second) </ins> } } }

I'm not using publicaddressoverride. At boot I'm trying to launch using: /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcdservers=http://10.13.137.165:4001 --address=0.0.0.0 --port=8080 --kubeletport= --allowprivileged=true --portalnet=10.254.0.0/16 But this gets launched before eth0 has an ip address. This used to work. Now apiserver just refused to start (complaining about 127.0.0.1 being invalid). I think it needs to just keep trying until it gets an address and can get to work....
Adding --publicaddressoverride= as suggested doesn't help.... Nov 05 12:43:55 localhost kube-apiserver[713]: F1105 12:43:55. ] listen tcp 10.13.137.165:7080: bind: cannot assign requested address Which is a fatal error. I can't bind to it. It isn't up yet.
Bisect says either commit or
If publicaddressoverride is specified, I can put the read-only server bind in a goroutine and retry periodically. If it's not, though, since we can't change the public address once started, we'd have to delay construction until a suitable address shows up. That's pretty ugly; is it an important case to support?
I personally think delaying construction is fine. Being able to launch the apiserver on any node from the init system and have it just work 99% of the time seems like a good thing....
OK. Fix coming in 3, 2, 1...
See if does the trick for you.
commented there. it fixed it.


Question: # Start kube-apiserver # omit enums from static openapi snapshots used to generate clients until #109177 is resolved <ins> # TODO(aojea) remove ConsistentListFromCache after https://issues.k8s.io/123674 </ins> kube::log::status "Starting kube-apiserver" kube-apiserver  --bind-address="${API_HOST}"

Verify Master - Master Blocking verify.openapi-spec Inconsistently since 2 November. First failure (shown in testgrid) API Server is not becoming ready during Relevant logs /sig-testing
cc for cidr controller
and may be related.
/sig auth network
/remove-sig auth Is almost certainly related
/assign it seems the servicecidr controller takes time to start and it blocks the apiserver readiness
/reopen Test run failed after related PR merger:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yeah /assign
/triage accepted
this is sig network can you pelase update the title?
/retitle [Flaking Test][sig-network] ci-kubernetes-verify-master verify-openapi-spec
This still flake in master blocking test grid.
use 60s timeout and here uses 30s timeout by default. It flakes. 10 lines of log cannot show other details. My local test with 15s timeout failed several times. 30s is enough for my local tests. Or we just enlarge the timeout for CI.
having both scripts the same timeout sounds reasonable I'm still worried why it keeps timing out, but for that we should first define a test that measure the apiserver startup and then decide what value is correct and enforce it.
/priority important-soon /milestone v1.30 backport to v1.29 may be needed.
Hello! Release Signal shadow here. This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting (less than a month from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this issue is tagged for 1.30, is it still planned for this release?
This flakes in Do we have some action items for v1.30?
I need to fix it, is a shame on me, sorry
Hello! Release Signal shadow here. This issue has not been updated for a long time, so we'd like to check the status. The code freeze is starting less than a week, and we want to ensure that each PR has a chance to be merged on time.

As this issue is tagged for 1.30, is it still planned for this release? Please let us know the status if possible.
yeah, I like to get it fixed, there is some race condition that makes some of the boostrap to deadlock , I still didn't find why
didn't forget but didn't have time to figure out the root cause, will try to get back to this end of the week
I got one failure locally of an apiserver that took much time The apiservers need the servicecidr controller to start to be healthy, and the controllers need to sync the informer, but checking one of the requests from the informer I can see this error logging error output: "k8sx00nfnx02v1x12x06Statusx12x96x01nx06nx00x12x00x1ax00x12aFailurex1a2Timeout: Too large resource version: 2, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" and this keep going for a while logging error output: "k8sx00nfnx02v1x12x06Statusx12x98x01nx06nx00x12x00x1ax00x12aFailurex1a4Timeout: Too large resource version: 203, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" do you know why the resource version is growing?
looks like the server returns that error in a few places, not sure which one is getting exercised: update-openapi-spec starts with AllAlpha=true in order to enable all features / APIs ... are there any alpha features around watch / resourceVersion that could cause this?
Didn't read carefully the issue, so just quickly sharing some thoughts: 1) streaming lists might potentially be related, but it needs to be explicitly requested by the client.

Reflector is the only place that can currently do that, but it requires explicitly set env-var: Is it possible that this value is set? if so, then streaming lists will result is calling etcd for current RV and will wait for watchache to be synced with it - if progress-notify-interval is large and manual progress notify is disabled, we could face things like that ^^ 2) that said, if Alpha features are enabled, then manual-progress notify should also be set: 3) is it possible that we're facing the bug that Marek found 2 days before release that effectively was the main reason we didn't try to promote consistnet read to beta? ^^
Based on the log from I would guess this is the 3002ms implies the 3 second timeout of waiting for revision Empty implies consistent list from cache implies non-recursive list Planned to fix it, but PR was blocked on
thanks a ton for digging into that to deflake this test, I'd set this in the update-openapi- file (since it doesn't impact the generated openapi) with a comment to remove once is resolved


Question: `node_ip` | (Optional) The IP address to use to address this node `minion_ip` | (Optional) Mapped to the kubelet hostname_override, K8S TODO - change this name `network_mode` | (Optional) Networking model to use among nodes: *openvswitch* <ins> `networkInterfaceName` | (Optional) Networking interface to use to bind addresses, default value *etho0* `publicAddressOverride` | (Optional) The IP address the kube-apiserver should use to bind against for external read-only access </ins> `roles` | (Required) 1. `kubernetes-master` means this machine is the master in the kubernetes cluster.  2. `kubernetes-pool` means this machine is a kubernetes-minion.  Depending on the role, the Salt scripts will provision different resources on the machine. These keys may be leveraged by the Salt sls files to branch behavior.

The kubelet.service EnvironmentFile /etc/sysconfig/kubelet doesn't exist on my minions.
Some extra information from master and minion-1:
I didn't think we used pki with salt. I wonder why salt is complaining about it. , David Reynolds wrote:
Is it possible this is related to salt's tar option having changed? A fix for that went in earlier today, but I don't know that we fixed vagrant yet...
Something else when I try to ./cluster/kube- This happens for each minion.
I think it's looking for eth0 in that ips list but isn't finding it?
since you're in the maintainers file for vagrant: any ideas?
- not yet, but I can take a look
- just to help debug, was this a fresh cluster you created that was working and then stopped after a push, or was this a fresh cluster that never worked?
It was a cluster that was working, I did a push and it broke. So I tore it down completely and rebuilt a fresh cluster that doesn't work either.
I am actually getting an issue where if I run more than one minion, my salt-minion is no longer able to contact the salt-master. So there is a number of things I need to look into.
So eth0 does not exist on a vagrant box for me, we look to need to use eth1 in a vagrant env.
appears to have introduced the regression on the vagrant env. is causing the issue.
Looking to see if I can specify which network interface to use for the function so we can safely use ips[0][0] when in an environment where the interface is not eth0
Just synced my branch, and found the issue is not limited to vagrant. When I ran hack/dev-build-kube-, I have the same kubeletservicecannotrestart issue due to lack of /var/lib/kubelet/kubernetesauth: But kubernetesauth is not installed: root ls /var/lib/kubelet/kubernetesauth ls: cannot access /var/lib/kubelet/kubernetes_auth: No such file or directory cc/
There are a number of issues, the master is not picking up the right network interface to use for the kube-apiserver to serve the readonly content, think we need a entry for address and publicAddressOverride per the arguments to kube-apiserver.

I don't have much more time to debug this tonight, but there are a number of issues we need to resolve here.
I ran kube- to recreate the cluster, and now /var/lib/kubelet/kubernetesauth is properly installed on nodes, and kubelet is succussfully started without any issue. Looks like we ran into saltcannotupgradeall_files issue we saw before.
- I do not see same behavior on a fresh cluster.
I am referring to gce, not vagrant. I think there are more issues on vagrant. But on gce, fresh cluster don't have issue.
I think I am close to a fix for vagrant.
Submitted a fix. Testing appreciated.
Bringing up a new cluster on vagrant. Will report back in a bit.
Works for me! Fresh cluster build
Works for me too. Awesome work. Thanks!
Derek, do you folks run vagrant e2e? If not maybe I should add that to our jenkins e2e box... --brendan , David Reynolds wrote:
Brendan, If you are willing to add it to your jenkins box that would be appreciated. -- Derek ----- Original Message ----- From: "Brendan Burns" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Wednesday, November 19, 2014 11:45:46 PM Subject: Re: [kubernetes] Kubelet not starting () Derek, do you folks run vagrant e2e? If not maybe I should add that to our jenkins e2e box... --brendan , David Reynolds wrote: Reply to this email directly or view it on GitHub:
The problem with upgrading the cluster is that it references a new file that doesn't exist if you are doing an upgrade. I'm going to check in a default version of that file to make this work and to smooth things over when the check fails.


Question: return err } visitor := o.ResourceFinder.Do() <del> if visitor, ok := visitor.(*resource.Result); ok && strings.ToLower(o.ForCondition) == "delete" { </del> <ins> isForDelete := strings.ToLower(o.ForCondition) == "delete" if visitor, ok := visitor.(*resource.Result); ok && isForDelete { </ins> visitor.IgnoreErrors(apierrors.IsNotFound) }

What happened: given an empty namespace (no pods), if running: , kubectl errors with the message . What you expected to happen: kubectl should exit with status 0, since the condition to wait for is for all pods to be deleted, and that condition is already satisfied. How to reproduce it (as minimally and precisely as possible): run the example kubectl command Anything else we need to know?: Environment: Kubernetes version (use ): Taking the liberty of skipping other version information, since this is (I assume) purely a client-side bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
Seems like didn't fix it correctly: Test in this PR always passed because the always return nil error and visitCount == 1, and the type conversion always fail here:
thanks for the quick fix!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale - still an issue, I believe the PR is waiting for re-review
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten

One use-case of is to wait for a lot of resources that you have just deleted. Unfortunately, this workflow is racy because will fail when encountering a resource that doesn't exist (yes, it's probably already gone). I suspect we should just consider 404 resources to be already gone and integrate that condition as part of the normal flow. /sig cli /kind bug
/assign
Duplicate of ?
yes, I tried to add in in But that PR seems not complete.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
The PR merged, I think we're good now!

Hi. A similar issue was reported here: but the bot had closed it due to inactivity. So, creating a new one with details again. I am facing the same problem on my end. If a resource does not exist or is deleted, but you are using wait --for condition delete, it fails. There is a flag in delete to avoid such errors. A similar condition of flag must be in as well so that it does not fail with errors like below. Thanks.
/sig cli
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Sorry I forget to do this, I'll try to submit a PR this week.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Yes, this would be nice to have.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Any chance this can be revisited? Thanks!


Question: // Wait for the pod to go away and stop monitoring once it does // TODO (jdefelice) replace with an /events watch? for { <del> time.Sleep(containerPollTime) </del> <ins> time.Sleep(lostPodPollTime) </ins> if k.checkForLostPodTask(driver, taskId, knownPod) { return }

In , there is a local constant named containerPollTime that is put to two very different uses --- waiting for a pod to come up, and recovering from pods getting "lost". The former needs to be very responsive, but the latter does not (but is still costly when the number of pods grows). The value is currently set to one second, and that makes both things happen frequently. It would be better if there were two different constants, so the monitoring for lost pods can be less frequent.


Question: <del> # How to use IPVS </del> <ins> # IPVS </ins> <del> This document shows how to use kube-proxy ipvs mode. </del> <ins> This document intends to show users - what is IPVS - difference between IPVS and IPTABLES - how to run kube-proxy in ipvs mode and info on debugging </ins> ## What is IPVS

/kind documentation What happened: IPVS needs to be documented more. Specifically, the existing documentation needs to be updated to properly describe how to bring up a cluster with ipvs in the kubeadm, local and gce case. Furthermore, there needs to be another piece of documentation that describes the main differences between iptables and ipvs and also dives into implementation differences between the two (ex. ipset). It would also be really nice if there could be some screenshots detailing the difference between iptables and ipvs rules (i.e bring up a cluster, create a service, dump iptables and ipvs rules, compare) All this documentation should be to pkg/proxy/ipvs /assign /area ipvs /sig network


Question: e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" "k8s.io/kubernetes/test/e2e/network/common" <del> gcecloud "k8s.io/legacy-cloud-providers/gce" </del> admissionapi "k8s.io/pod-security-admission/api" netutils "k8s.io/utils/net" utilpointer "k8s.io/utils/pointer"

master-informing: ! 2023-09-07 03:13:59 +0000 UTC No response /sig network
Checking this failure This is running ingress-gce version, logs in failure is It seems the healthceck interval is not updated correctly, by reading the test tagging the people with more expertise in this area /cc
/cc
it seems the test restarts the kube-controller-manager but we landed recently the switch from intree to outtree cloud providers, so this logic now runs in the cloud-controller-manager, can this be the problem, that we have to restart the cloud-controller-manager now instead of the kcm?
/triage accepted This flake started after and kept failing after 09-07.
yeah, the logic that handles the loadbalancer shifted to that cloud controller manager with that PR


Question: ], ) <del> go_test( name = "go_default_test", srcs = [ "cluster_test.go", "configmap_test.go", "deployment_test.go", "generate_test.go", "kubectl_test.go", "namespace_test.go", "proxy_server_test.go", "quota_test.go", "rolebinding_test.go", "rolling_updater_test.go", "rollout_status_test.go", "run_test.go", "scale_test.go", "secret_for_docker_registry_test.go", "secret_for_tls_test.go", "secret_test.go", "service_basic_test.go", "service_test.go", "serviceaccount_test.go", "sorting_printer_test.go", "stop_test.go", ], library = ":go_default_library", tags = ["automanaged"], deps = [ "//federation/apis/federation/v1beta1:go_default_library", "//pkg/api:go_default_library", "//pkg/api/testapi:go_default_library", "//pkg/api/testing:go_default_library", "//pkg/api/v1:go_default_library", "//pkg/apis/apps:go_default_library", "//pkg/apis/apps/v1beta1:go_default_library", "//pkg/apis/batch:go_default_library", "//pkg/apis/batch/v1:go_default_library", "//pkg/apis/batch/v2alpha1:go_default_library", "//pkg/apis/extensions:go_default_library", "//pkg/apis/extensions/v1beta1:go_default_library", "//pkg/apis/rbac:go_default_library", "//pkg/client/clientset_generated/internalclientset:go_default_library", "//pkg/client/clientset_generated/internalclientset/fake:go_default_library", "//pkg/client/clientset_generated/internalclientset/typed/batch/internalversion:go_default_library", "//pkg/client/clientset_generated/internalclientset/typed/core/internalversion:go_default_library", "//pkg/client/clientset_generated/internalclientset/typed/extensions/internalversion:go_default_library", "//pkg/kubectl/util:go_default_library", "//vendor/github.com/spf13/cobra:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/equality:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/resource:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured:go_default_library", "//vendor/k8s.io/apimachinery/pkg/runtime:go_default_library", "//vendor/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/intstr:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/uuid:go_default_library", "//vendor/k8s.io/apimachinery/pkg/watch:go_default_library", "//vendor/k8s.io/client-go/rest:go_default_library", "//vendor/k8s.io/client-go/rest/fake:go_default_library", "//vendor/k8s.io/client-go/testing:go_default_library", "//vendor/k8s.io/client-go/util/testing:go_default_library", ], ) </del> filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], <ins> visibility = ["//visibility:private"], </ins> ) filegroup(

StatefulSet now as of 1.6. That means the TODO on can be fixed in 1.7. We could not fix it in 1.6 because that code is used by kubectl, which must work against servers that are one minor version ahead of or behind the client.
/cc
I am volunteered to take this job if no one start working on this.
I don't think anyone's working on it, so you are welcome to take it. Feel free to tag me as a reviewer. Thanks!


Question: if test.podStatus != nil { m.statusManager.SetPodStatus(w.pod, *test.podStatus) } <del> if c := w.doProbe(); c != test.expectContinue { t.Errorf("[%s-%d] Expected continue to be %v but got %v", probeType, i, test.expectContinue, c) </del> <ins> if test.setDeletionTimestamp { now := metav1.Now() w.pod.ObjectMeta.DeletionTimestamp = &now } if c := w.doProbe(); c != test.expectContinue[probeType.String()] { t.Errorf("[%s-%d] Expected continue to be %v but got %v", probeType, i, test.expectContinue[probeType.String()], c) </ins> } result, ok := resultsManager(m, probeType).Get(testContainerID) if ok != test.expectSet {

What happened: liveness/readiness probe fails while pod is terminated. Also it happened only once during the pod termination. The issue started happening after upgrading version to v1.7 from v1.6.X How to reproduce it (as minimally and precisely as possible): execute to delete pod, so status of the nginx-podA1 is changed to , right after that it seems Liveness and Readiness Probe is executed and failed, but only once. Nginx reverse proxy is running in the pod. so I just use method for liveness and readiness Here is my Deployment config. Here is Events log by Environment: Kubernetes version: 1.7.2
/kind bug
/sig node
PING
Since the upgrade to 1.7, it seems our deployment rollouts have a higher failure rate. Occasionally, the pod would come up, but no readiness probe ever gets started. It stays in that state, blocking the entire deployment. I usually have to delete the pod so it is rescheduled, and a new readiness probe is fired to check. I wonder if these are related issues here.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen /remove-lifecycle rotten We see this consistently with all pods who define a liveness or readiness probe. Whenever we roll out a new deployment, the pods who are terminated will emit a failed liveness/readiness probe AFTER they have been terminated.

We have considered adding a hook that just sleeps for 2-3 seconds, but it seems like a band-aid solution to something that should not happen in the first place. Is this an impossible-to-solve race condition between kubernetes moving parts?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Do you still experience this? This happens to us running kubernetes 1.11.4.
On k8s 1.12.7 and this is happening to me as well.
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} still happen. I think just add a phase "Terminateing" in pod, when an pod is in Terminateing just stop check.
I think it still happening in 1.13, why this issue was closed? Logs, !
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
still experience this on kubernetes 1.12.4. delete event and probe failed event so it seems that after the pod is deleted, the probe keeps running.I am wondering if the probe is started before the pod deletion.
/area kubelet
/priority important-soon
Any progress on this issue? It seems like some kind bug.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /assign
GitHub didn't allow me to assign the following users: ashleyschuett. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /remove-lifecycle rotten /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
16.9 here. Same issue.
/assign
I am also facing this issue. We have pods which have a lot of cleanup to do during shutdown, it can take up to 5 mins to terminate gracefully. During this time the livelinessProbe is detecting failure and restarting the pod. not really what we want. I am unable to prevent the service that handles the liveliness check from stopping while the cleanup is happening. It would be better if the pod was immediately removed from the service and the probes stopped while the shutdown is performed. Basically this ends up that k8s never actually is able to terminate the pod.
I will take that point and propose a shutdown probe to API sig.

Same here at 1.18.8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
kubernetes_version 1.19.3. Same issue.
Same here, kubernetes 1.19.4.
/remove-lifecycle rotten
/assign
Should I consider a cherry-pick for 1.20 and 1.19? (maybe 1.18 too?)
we're running 1.16 and being hit by this continuously when some of our elixir apps shutdown so cherry-picking it in 1.18 would at least put it closer in our update path :angel:


Question: rc = int(ee.Sys().(syscall.WaitStatus).ExitStatus()) Logf("rc: %d", rc) } <del> return stdout.String(), uexec.CodeExitError{ </del> <ins> return stdout.String(), stderr.String(), uexec.CodeExitError{ </ins> Err:  fmt.Errorf("error running %v:nCommand stdout:n%vnstderr:n%vnerror:n%v", cmd, cmd.Stdout, cmd.Stderr, err), Code: rc, } } case <-b.timeout: b.cmd.Process.Kill() <del> return "", fmt.Errorf("timed out waiting for command %v:nCommand stdout:n%vnstderr:n%v", cmd, cmd.Stdout, cmd.Stderr) </del> <ins> return "", "", fmt.Errorf("timed out waiting for command %v:nCommand stdout:n%vnstderr:n%v", cmd, cmd.Stdout, cmd.Stderr) </ins> } Logf("stderr: %q", stderr.String()) Logf("stdout: %q", stdout.String()) <del> return stdout.String(), nil </del> <ins> return stdout.String(), stderr.String(), nil </ins> } // RunKubectlOrDie is a convenience wrapper over kubectlBuilder

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-windows jobs: Which test(s) are failing: [sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance] [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] Since when has it been failing: It was a false positive until recently. Testgrid link: Reason for failure: Partially Qualified Domain Names are not currently supported on Windows containers. The tests are checking PQDNs for validation. The test was passing until recently because was still returning the exit code 0, even if the given DNS name was not found: [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] [BeforeEach] [sig-network] Services STEP: Creating a kubernetes client Apr 28 21:24:38.587: INFO: kubeConfig: /home/ubuntu/kube/config200424 STEP: Building a namespace api object, basename services STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in services-2633 STEP: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-network] Services [It] should be able to change the type from NodePort to ExternalName [Conformance] STEP: creating a service nodeport-service with the type=NodePort in namespace services-2633 STEP: Creating active service to test reachability when its FQDN is referred as externalName for another service STEP: creating service externalsvc in namespace services-2633 STEP: creating replication controller externalsvc in namespace services-2633 I0428 21:24:42. ] Created replication controller with name: externalsvc, namespace: services-2633, replica count: 2 I0428 21:24:45. ] externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:48. ]

externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:51. ] externalsvc Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:54. ]

externalsvc Pods: 2 out of 2 created, 2 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady STEP: changing the NodePort service to type=ExternalName Apr 28 21:24:55.544: INFO: Creating new exec pod Apr 28 21:25:06.371: INFO: Running '/home/ubuntu/rune2e/kubernetes/output/dockerized/bin/linux/amd64/kubectl --server=https://kubetest--83ef-11ea-ae60- --kubeconfig=/home/ubuntu/kube/config200424 exec --namespace=services-2633 agnhost-pod -- /bin/sh -x -c nslookup nodeport-service' Apr 28 21:25:09.610: INFO: stderr: "+ nslookup nodeport-servicen kube- can't find nodeport-service: Non-existent domainrrn" Apr 28 21:25:09.610: INFO: stdout: "Server: kube-rnAddress: 10.0.0.10rnrn" STEP: deleting ReplicationController externalsvc in namespace services-2633, will wait for the garbage collector to delete the pods Apr 28 21:25:10.429: INFO: Deleting ReplicationController externalsvc took: 423.628373ms Apr 28 21:25:10.830: INFO: Terminating ReplicationController externalsvc pods took: 400.686463ms Apr 28 21:25:15.792: INFO: Cleaning up the NodePort to ExternalName test service [AfterEach] [sig-network] Services Apr 28 21:25:16.230: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready STEP: Destroying namespace "services-2633" for this suite. [AfterEach] [sig-network] Services â¢{"msg":"PASSED [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance]","total":2,"completed":2,"skipped":2913,"failed":0} The issue was observed with an updated, which doesn't return 0 if the given DNS name was not found. Anything else we need to know*: /sig testing /sig windows


Question: var deletedNodeName string for _, originalNode := range originalNodes.Items { originalNodeName := originalNode.ObjectMeta.Name <ins> var found bool </ins> for _, targetNode := range targetNodes.Items { if originalNodeName == targetNode.ObjectMeta.Name { <del> continue </del> <ins> found = true break </ins> } } <del> deletedNodeName = originalNodeName break </del> <ins> if !found { deletedNodeName = originalNodeName break } </ins> } gomega.Expect(deletedNodeName).NotTo(gomega.BeEmpty()) gomega.Eventually(ctx, func() error {

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig cloud-provider-gcp testing
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : ### Failure cluster /kind failing-test /sig cloud-provider-gcp testing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Take as an example. In log, we can see the waiting for a node which is not deleted. test--minion-group-99m3 was the first node and not deleted in the node lease e2e test test--minion-group-dhj9 is the node that was removed in the e2e test.
/cc