Question: ) destChain := svcXlbChain <del> // We have to SNAT packets to external IPs if externalTrafficPolicy is cluster. </del> <ins> // We have to SNAT packets to external IPs if externalTrafficPolicy is cluster // and the traffic is NOT Local. Local traffic coming from Pods and Nodes will // be always forwarded to the corresponding Service, so no need to SNAT // If we can't differentiate the local traffic we always SNAT. </ins> if !svcInfo.OnlyNodeLocalEndpoints() { destChain = svcChain <del> writeLine(proxier.natRules, append(args, "-j", string(KubeMarkMasqChain))...) </del> <ins> // This masquerades off-cluster traffic to a External IP. if proxier.localDetector.IsImplemented() { writeLine(proxier.natRules, proxier.localDetector.JumpIfNotLocal(args, string(KubeMarkMasqChain))...) } else { writeLine(proxier.natRules, append(args, "-j", string(KubeMarkMasqChain))...) } </ins> } <ins> // Sent traffic bound for external IPs to the service chain. writeLine(proxier.natRules, append(args, "-j", string(destChain))...) </ins> <del> // Allow traffic for external IPs that does not come from a bridge (i.e. not from a container) // nor from a local process to be forwarded to the service. // This rule roughly translates to "all traffic from off-machine". // This is imperfect in the face of network plugins that might not use a bridge, but we can revisit that later. externalTrafficOnlyArgs := append(args, "-m", "physdev", "!", "--physdev-is-in", "-m", "addrtype", "!", "--src-type", "LOCAL") writeLine(proxier.natRules, append(externalTrafficOnlyArgs, "-j", string(destChain))...) dstLocalOnlyArgs := append(args, "-m", "addrtype", "--dst-type", "LOCAL") // Allow traffic bound for external IPs that happen to be recognized as local IPs to stay local. // This covers cases like GCE load-balancers which get added to the local routing table. writeLine(proxier.natRules, append(dstLocalOnlyArgs, "-j", string(destChain))...) </del> } else { // No endpoints. writeLine(proxier.filterRules,

What would you like to be : e2e test for service ExternalIPs Why is this needed: There is no test covering Service ExternalIPs Since we can not assume that the e2e binary will have connectivity against the nodes and that we can add routes to the environment, we are going to test it from a pod. I've verified that this works locally with the following commands: The test should leverage current framework, maybe something as A good example is here, maybe adding a ?
/sig network /help wanted /triage accepted
/help wanted /good-first-issue
/assign
I'd like to pick this up!
Ok,I'm not started :) /unassign /assign
There are two different externalIP codepaths, one for the case where the claimed external IP exists as an IP on the default interface, and one for where it does not. I suppose we could just temporarily add an IP to the default interface...
good idea Dan Please remember, it has to work for IPv6 and for IPv4 ... anyway, we can address these things during the review
This has been partially implemented with . Covering the code path for when an IP is does not exist on a node. I'll assign this back to you so you can split this issue or close it in favor of another issue. Thanks! /unassign /assign
what is the different codepath for ExternalIPs that already exist on the host? ~My reading is that the only difference is that it opens a local socket on that port, can you expand on this?~ Found it
so, it seems that this work in some environments because they are not using the bridge plugin why is not allowed that pods can reach ExternalIPs?
Wow, this is ancient and needs some fixing. This is not about preventing pod traffic from reaching those IPs. I had to really stare at it, and I am not 100% in my interpretation. First, we recognize such traffic from "off machine" (!physdev-is-in && !src=LOCAL, which we should probably switch to localDetector) and steer it to the Service. Second, we recognize ANY such traffic that is destined for a LOCAL address and shortcut that to the Service, Implicitly, traffic destined for this IP where the IP is not local goes to the system's default routing logic.

The reliance on the bridge plugin is awful and archaic and I will buy a beverage of your choice for whomever gets rid of that. I think localDetector can cover this, but not 100% sure
but the ExternalIP belongs to a service and we(kube-proxy) "knows" where to send it (to the service) , that is the part I'm not sure to understand, because whatever the user is doing with that ExternalIP it has to send it back to the cluster to reach the service. However, there is always context that is important and maybe I'm missing something. Assuming we want to handle ALL the ExternalIP traffic with kube-proxy, the logic will be much simpler: traffic to an externalIP will be ALWAYS sent to the Service (DNAT) if traffic is NOT LOCAL, we also SNAT
/remove-good-first-issue /remove-help-wanted


Question: // Let the controller-manager allocate Node CIDRs for the Pod network. // Each node will get a subspace of the address CIDR provided with --pod-network-cidr. if cfg.Networking.PodSubnet != "" { <del> maskSize := "24" if ip, _, err := net.ParseCIDR(cfg.Networking.PodSubnet); err == nil { if ip.To4() == nil { maskSize = "64" } } </del> <ins> maskSize := calcNodeCidrSize(cfg.Networking.PodSubnet) </ins> command = append(command, "--allocate-node-cidrs=true", "--cluster-cidr="+cfg.Networking.PodSubnet, "--node-cidr-mask-size="+maskSize) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: For IPv6, the node CIDR size is hard coded to /64. This causes an error bringing up the kube-controller, as the size must be for a smaller subnet (larger value) that what is used for the pod subnet, which is specified for some CNI plugins (e.g. Calico). In addition, this is a catch-22, as the pod subnet prefix cannot be for a larger network (smaller value) than /66, so we cannot set it to /63, for example. What you expected to happen: The node CIDR size should be computed, based on the pod subnet specified, so that it can meet the requirements. How to reproduce it (as minimally and precisely as possible): Bring up a cluster using KubeAdm, specifying IPv6 and choose any pod subnet prefix desired. Anything else we need to know?: Note, there is a second requirement, that the difference between pod subnet size and node CIDR size be 16 or less (issue ). I'll be submitting a proposed PR to address this. Environment: Kubernetes version (use ): latest on master, using 1.9.2 in my case Cloud provider or hardware configuration: bare metal, single node OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux bxb-c2-77 4.4.0-109-generic -Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others: /area ipv6 /sig network
/assign pmichali


Question: } // Act <del> err = asw.MarkDeviceAsMounted(generatedVolumeName) </del> <ins> err = asw.MarkDeviceAsMounted(generatedVolumeName, devicePath) </ins> // Assert if err != nil {

We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by


Question: n.Labels = map[string]string{} } <del> k8sNamespaceRegex := regexp.MustCompile("(kubernetes|k8s).io/") </del> <ins> k8sNamespaceRegex := regexp.MustCompile(`(^|.)(kubernetes|k8s).io/`) </ins> for k, v := range instanceMeta.AdditionalLabels { // Cloud provider should not be using kubernetes namespaces in labels if isK8sNamespace := k8sNamespaceRegex.MatchString(k); isK8sNamespace {

For cloud provider nodes, there are some k8s label names are reserved. Like and The current regex is: This will match the dot . as any single character. This block adding a label name like: app.k8snio/name The regex format is incorrect, we should escape the dot in regex so that it can match the dot as a string. Can add additional label with name like: app.k8snio/name Configure AdditionalLabels in the instanceMeta to add additional label name with app.k8snio/name. No response <detailsClient Version: v1.30.1 Kustomize Version: v5.0.4-0.- Server Version: v1.30.0 </details<details</details<details</details<details</details<details</details<details</details
/sig cloud-provider
If it make sense to fix it, i would like to take it and raise a PR. /assign
Which Kubernetes component has the bug ? (eg: kubelet) Can you explain how to reproduce the problem?
It's the node controller in the shared library Though I really don't think cloud providers should be setting node labels like "", this is technically incorrect. This restriction is only applied to labels the cloud provider controller binary running this controller would attempt to add itself.
just noticed this after our SIG meeting today, cc
/triage accepted
please go ahead and raise a PR, thanks for reporting this =)
When fixing this, consider if there are any labels that cloud provider integration might legitimately set. For example: (documented as being set by cloud provider integration code). Some labels are OK to set; we could allowlist and explicitly. When rejecting some others, maybe setting a Condition on the Node ( perhaps) would help surface what's not right. Not sure. Making the simple fix is also fine by me.
OK. Thanks I have raised a PR for this: Not sure if this change looks good.


Question: if [[ "${KUBERNETES_PROVIDER}" == "gke" ]]; then detect-node-instance-groups <del> NODE_INSTANCE_GROUP=$(kube::util::join , NODE_INSTANCE_GROUPS) </del> <ins> NODE_INSTANCE_GROUP=$(kube::util::join , "${NODE_INSTANCE_GROUPS[@]}") </ins> fi ginkgo_args=()

release-master-blocking: gce-device-plugin-gpu-masterci-kubernetes-e2e-gce-device-plugin-gpu.Overall kubetest.Up 3rd November 2022 Up | 28m51s -- | -- { error during ./hack/e2e-internal/e2e- exit status 1} No response /sig cloud-provider cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from:
we'll need to bump the node image i think:
Latest run is passing now!


Question: "github.com/google/gofuzz" ) <ins> func TestConverter_byteSlice(t *testing.T) { c := NewConverter() src := []byte{1, 2, 3} dest := []byte{} err := c.Convert(&src, &dest, 0, nil) if err != nil { t.Fatalf("expected no error") } if e, a := src, dest; !reflect.DeepEqual(e, a) { t.Errorf("expected %#v, got %#v", e, a) } } </ins> func TestConverter_DefaultConvert(t *testing.T) { type A struct { Foo string

E.g., it calls Convert() for every byte of a slice of uint8. E0330 22:15:23. 5352 ] TIM: converting bool E0330 22:15:23. 5352 ] TIM: converting []v1.NamedExtension E0330 22:15:23. 5352 ] TIM: converting []v1.NamedCluster E0330 22:15:23. 5352 ] TIM: converting []v1.NamedCluster E0330 22:15:23. 5352 ] TIM: converting v1.Cluster E0330 22:15:23. 5352 ] TIM: converting v1.Cluster E0330 22:15:23. 5352 ] TIM: converting string E0330 22:15:23. 5352 ] TIM: converting string E0330 22:15:23. 5352 ] TIM: converting bool E0330 22:15:23. 5352 ] TIM: converting string E0330 22:15:23. 5352 ] TIM: converting []uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23.

5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 ... thousands of these.
There's an easy fix for this :/


Question: } else { managed.Fields()[manager] = vs } <del> // Update the time only if the manager's fieldSet has changed. if formerSet == nil || !managed.Fields()[manager].Set().Equals(formerSet.Set()) { managed.Times()[manager] = &metav1.Time{Time: time.Now().UTC()} } </del> <ins> managed.Times()[manager] = &metav1.Time{Time: time.Now().UTC()} </ins> } return object, managed, nil

I created an object with , and later I updated the same object with another . After that I looked at the object's managedFields, and saw nothing about the update. Here is what I got after the second apply: I expected managedFields to get updated by the second . No response <details</details<detailsnone </details<details</details<detailskubeadm </details<detailsdocker </details<details</details
/cc
I also saw the timestamp not updated when I updated using client-go and typed client stubs, Apply method, patch type .
/assign /triage accepted
/cc
The is not updated because you are not changing the structure of the object (adding or removing fields). According to , the time is updated if and only if the fieldSet is changed. It does not take into account the value of fields. One option would be to not compare the fieldSets but directly compare the liveObject & newObject for differences and update the field accordingly. Is it the desired behavior to update the time of when only the value is updated?
Yes! Look at the first paragraph of .
is that documentation correct? Is this what we intended?
I don't see any mention to the timestamp in that first paragraph. We've clarified recently the exact semantic of the timestamp, as defined here: I think you've identified the wrong code if you want to go ahead and fix it, go ahead, I'll happily review!


Question: } } <del> var extraPods *v1.PodList var volumeAttachments *storagev1.VolumeAttachmentList var pvs *v1.PersistentVolumeList var nodes *v1.NodeList </del> func CreateTestClient() *fake.Clientset { <ins> var extraPods *v1.PodList var volumeAttachments *storagev1.VolumeAttachmentList var pvs *v1.PersistentVolumeList var nodes *v1.NodeList </ins> fakeClient := &fake.Clientset{} extraPods = &v1.PodList{}

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --
/sig storage /triage accepted
Pr continue failed beacuse this data race, and i did an fix in ,it's ok now.
/assign
another instance
another instance: (PR: )
flaked again since was merged
fixed in
/reopen Can you backport to 1.21?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
opened
Hit this bug also while working on a fix for 1.19 - so maybe we should open another issue/pr for 1.20 and 1.19 ? At least 1.19 will be still around for quite a while
I think that would probably be approved. if anyone wants to take a stab at it
Thanks I'll check over weekend if i can provide this fix also to 1.20 and 1.19
Opened for 1.19 Opened for 1.20
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nil, true, nil } <ins> // matchOnExitCodes returns a terminated container status that matches the error code requirement, if any exists. // If the returned status is non-nil, it has a non-nil Terminated field. </ins> func matchOnExitCodes(podStatus *v1.PodStatus, requirement *batch.PodFailurePolicyOnExitCodesRequirement) *v1.ContainerStatus { if containerStatus := getMatchingContainerFromList(podStatus.ContainerStatuses, requirement); containerStatus != nil { return containerStatus

Since the last days there is an unusual rate of flakiness on kind jobs, but couldn't find a pattern ! While looking at one test failure and checking at the controller manager logs, there is a panic on the controller manager that makes the controller manager to restart and impacts on all the test that are running at that time. ` Lessons learned, we should not increase timeouts forever or these problems will go unnoticed, still, they are really hard to find and is not the first time that this happens, maybe Kubernetes is too good reconciling XD It will be great to have a tool that detect panics on the components /sig scheduling
/release blocker /priority important-soon /kind bug /cc
looks like the exit code matching doesn't handle a mix of terminated and unterminated containers, needs something like this:
big +1, a restart or panic detector on control plane components would be excellent
cc
this sounds like a nice project ^^^ but is better to check first with and , I see there are some scripts that parse and dump the logs here maybe we can add some thing to report them somehow?
/assign
oops, it's holiday in Poland /unassign /assign
/remove-sig scheduling
I can add the fix proposed by but the underlying problem is . I'll prepare a PR for each and see if we are ok merging both or just the hotfix.
commented on we'll merge the panic fix right away in either case
Fix and unit test is up


Question: // Run "prebind" plugins. if status := fwk.RunPreBindPlugins(ctx, state, assumedPod, scheduleResult.SuggestedHost); !status.IsSuccess() { <ins> if status.IsRejected() { fitErr := &framework.FitError{ NumAllNodes: 1, Pod:         assumedPodInfo.Pod, Diagnosis: framework.Diagnosis{ NodeToStatusMap:      framework.NodeToStatusMap{scheduleResult.SuggestedHost: status}, UnschedulablePlugins: sets.New(status.Plugin()), }, } return framework.NewStatus(status.Code()).WithError(fitErr) } </ins> return status }

Currently, the scheduler honors Unschedulable(AndUnresolvable) or Pending only from PreFilter, Filter, Reserve, and Permit (WaitOnPermit). It doesn't expect other extension points to return those statuses and just regard all non-success status as errors, that is, immediately retrying scheduling of Pods. Actually returns Pending at PreBind, which is currently being ignored as described. So, we should expand UnschedulablePlugins/PendingPlugins to include PreBind plugins. (or maybe all extension points too?) /cc /sig scheduling /kind feature
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Both seems reasonable to me.
Because is not handled for , the following error is logged:
This patch gets rid of the error. I am uncertain whether it has other implications? Performance was the same with and without it.
What I imagined is like that. It shouldn't have any other impact because volumebinding, which is the only PreBind plugin apart from DRA in in-tree, only returns success or error status. We have to mention the change in the release note for custom plugin developers though.
I've turned this into a PR:


Question: ### netexec <del> Starts a HTTP server on given port with the following endpoints: </del> <ins> Starts a HTTP(S) server on given port with the following endpoints: </ins> - `/`: Returns the request's timestamp. - `/clientip`: Returns the request's IP address.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The image used in E2E tests is used for testing a wide variety of OS-agnostic features. One of these is , which runs an HTTP server with various endpoints that can be used to validate things like request/response. It would be useful to allow for 's functionality to also support HTTPS so all of that logic can be tested for TLS-based traffic. Currently, can be run like: I'm proposing something along the lines of: The default certificate/private key being basically identical to what provides. Why is this needed: One use case is where the intent is to add HTTPS functionality to lifecycle hooks (postStart/preStop). There is currently no easy way to test this logic because the container which receives the HTTPS request (in this case, ) does not support HTTPS traffic.
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing


Question: t.Fatal(err) } ss1Rev1.Namespace = ss1.Namespace <del> ss1Rev2, err := NewControllerRevision(ss1, parentKind, ss1.Spec.Template.Labels, rawTemplate(&ss1.Spec.Template), 2, ss1.Status.CollisionCount) </del> <ins> // Create a new revision with the same name and hash label as an existing revision, but with // a different template. This could happen as a result of a hash collision, but in this test // this situation is created by setting name and hash label to values known to be in use by // an existing revision. modTemplate := ss1.Spec.Template.DeepCopy() modTemplate.Labels["foo"] = "not_bar" ss1Rev2, err := NewControllerRevision(ss1, parentKind, ss1.Spec.Template.Labels, rawTemplate(modTemplate), 2, ss1.Status.CollisionCount) ss1Rev2.Name = ss1Rev1.Name ss1Rev2.Labels[ControllerRevisionHashLabel] = ss1Rev1.Labels[ControllerRevisionHashLabel] </ins> if err != nil { t.Fatal(err) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT: What happened: When a statefulset object is created, a lot of controllerrevisions are created at the same time. the controllerrevisions have differenct hash value but the same Data.Raw. How to fix it: two controllerrevisions are seen as the same , when [ControllerRevisionHashLabel] , Controllerrevision.Data.Raw , Controllerrevision.Data.Object are strictly the same // this line shoud be to fix the bug like below: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.8 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I would help fix it
/kind bug /sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
StatefulSet uses the following code to construct its history object (controllerrevision): However, collisionCount shouldn't be blindly increased when there's name collision. Because controller uses a cache store, the name collision could be due to a controllerrevision created but not yet observed by the controller. The right way to handle it is to first check if the controllerrevision with the same name has the same content as the one we're going to create, like how DaemonSet controller handles it:


Question: return available, capacity, usage, inodes, inodesFree, inodesUsed, nil } <del> // DiskUsage gets disk usage of specified path. func DiskUsage(path string) (*resource.Quantity, error) { // First check whether the quota system knows about this directory // A nil quantity with no error means that the path does not support quotas // and we should use other mechanisms. data, err := fsquota.GetConsumption(path) if data != nil { return data, nil } else if err != nil { return nil, fmt.Errorf("unable to retrieve disk consumption via quota for %s: %v", path, err) } // Uses the same niceness level as cadvisor.fs does when running du // Uses -B 1 to always scale to a blocksize of 1 byte out, err := exec.Command("nice", "-n", "19", "du", "-x", "-s", "-B", "1", path).CombinedOutput() if err != nil { return nil, fmt.Errorf("failed command 'du' ($ nice -n 19 du -x -s -B 1) on path %s with error %v", path, err) } used, err := resource.ParseQuantity(strings.Fields(string(out))[0]) if err != nil { return nil, fmt.Errorf("failed to parse 'du' output %s due to error %v", out, err) } used.Format = resource.BinarySI return &used, nil } </del> <ins> // DiskUsage calculates the number of inodes and disk usage for a given directory func DiskUsage(path string) (UsageInfo, error) { var usage UsageInfo </ins> <del> // Find uses the equivalent of the command `find <path> -dev -printf '.' | wc -c` to count files and directories. // While this is not an exact measure of inodes used, it is a very good approximation. func Find(path string) (int64, error) { </del> if path == "" { <del> return 0, fmt.Errorf("invalid directory") </del> <ins> return usage, fmt.Errorf("invalid directory") </ins> } // First check whether the quota system knows about this directory <del> // A nil quantity with no error means that the path does not support quotas // and we should use other mechanisms. inodes, err := fsquota.GetInodes(path) if inodes != nil { return inodes.Value(), nil } else if err != nil { return 0, fmt.Errorf("unable to retrieve inode consumption via quota for %s: %v", path, err) </del> <ins> // A nil quantity or error means that the path does not support quotas // or xfs_quota tool is missing and we should use other mechanisms. consumption, _ := fsquota.GetConsumption(path) if consumption != nil { usage.Bytes = consumption.Value() </ins> } <del> var counter byteCounter var stderr bytes.Buffer findCmd := exec.Command("find", path, "-xdev", "-printf", ".") findCmd.Stdout, findCmd.Stderr = &counter, &stderr if err := findCmd.Start(); err != nil { return 0, fmt.Errorf("failed to exec cmd %v - %v; stderr: %v", findCmd.Args, err, stderr.String()) </del> <ins> inodes, _ := fsquota.GetInodes(path) if inodes != nil { usage.Inodes = inodes.Value() </ins> } <del> if err := findCmd.Wait(); err != nil { return 0, fmt.Errorf("cmd %v failed. stderr: %s; err: %v", findCmd.Args, stderr.String(), err) </del> <ins> if inodes != nil && consumption != nil { return usage, nil </ins> } <del> return counter.bytesWritten, nil } // Simple io.Writer implementation that counts how many bytes were written. type byteCounter struct{ bytesWritten int64 } </del> <ins> topLevelStat := &unix.Stat_t{} err := unix.Stat(path, topLevelStat) if err != nil { return usage, err } </ins> <del> func (b *byteCounter) Write(p []byte) (int, error) { b.bytesWritten += int64(len(p)) return len(p), nil </del> <ins> // dedupedInode stores inodes that could be duplicates (nlink > 1) dedupedInodes := make(map[uint64]struct{}) err = filepath.Walk(path, func(path string, info os.FileInfo, err error) error { // ignore files that have been deleted after directory was read if os.IsNotExist(err) { return nil } if err != nil { return fmt.Errorf("unable to count inodes for %s: %s", path, err) } // according to the docs, Sys can be nil if info.Sys() == nil { return fmt.Errorf("fileinfo Sys is nil") } s, ok := info.Sys().(*syscall.Stat_t) if !ok { return fmt.Errorf("unsupported fileinfo; could not convert to stat_t") } if s.Dev != topLevelStat.Dev { // don't descend into directories on other devices return filepath.SkipDir } // Dedupe hardlinks if s.Nlink > 1 { if _, ok := dedupedInodes[s.Ino]; !ok { dedupedInodes[s.Ino] = struct{}{} } else { return nil } } if consumption == nil { usage.Bytes += int64(s.Blocks) * int64(512) // blocksize in bytes } if inodes == nil { usage.Inodes++ } return nil }) return usage, err </ins> }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: in calculates number of inodes wrong when there are hard links. What you expected to happen: inodes with hardlinks should only be counted once. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Issue is still not solved. There is an open PR waiting for review. /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
still waiting for to be merged

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to run pods on Alpine Linux with busybox find. What you expected to happen: I expected the call to exit with success, not failure. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing GNU findutils, so busybox implementation of is used. Anything else we need to know?: failed because the argument is not implemented in busybox . The option is not specified in so it can not be expected to be supported in all implementations. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related bug Others: I think it would be better to use instead of Related to
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This is solved by which is waiting for a review. /remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Still waiting for the PR to be merged. /remove-lifecycle rotten
/remove-lifecycle rotten

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running test suite on a cluster with workers on Alpine Linux, which uses busybox instead of GNU coreutils fails. The log shows that it fails to execute the command. What you expected to happen: I expected the call to exit with success. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing . Anything else we need to know?: It failed due to the option which is not implemented in busybox. The option is not specified in so I don't think it is reasonable to expect it to implemented in busybox. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related Others:
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale fixes this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten
Still waiting for PR to be merged


Question: "k8s.io/kubernetes/pkg/proxy/ipvs" ) <ins> // (I am unsure if this test has any value since it only tests the fake implementation) </ins> func TestSetGetLocalAddresses(t *testing.T) { fake := NewFakeNetlinkHandle(false) _ = ipvs.NetLinkHandle(fake) // Ensure that the interface is honored

Hey, after upgrading kubernetes from 1.22 to 1.23, I'm experiencing strange service provisioning behavior using nodePort. I'm using kube-proxy (ipvs) + calico + metallb. If external IP is used by another service type (Loadbalancer - allocated by metallb), then the node owning this IP doesn't listen to NodePort on this interface. Unfortunately, I can't find any changes that could be causing this, any ideas? Kubernetes 1.23 Missing externalip1 here: Using k8s 1.22 service with nodeport caused listening on every interface, despite using external IP by another service. Kubernetes 1.22 Upgrade k8s from 1.22.15 to 1.23.15 No response <details</details<detailson-premise instances </details<details</details<detailskubeadm </details<detailsdocker </details<detailscalico v3.24.5 metallb v0.13.7 ipset v7.5, protocol version: 7 ipvsadm v1.31 2019/12/24 (compiled with popt and IPVS v1.2.1) #ipvs config: </details
/sig network
/assign
From : This is not a valid setup. A virtual IP (VIP), like the loadBalancerIP, must not be assigned to a physical interface. The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without_ assigning the VIP to any interface.
When you send traffic to a nodePort, the destination address should be a node address, not a loadBalancerIP.
There is however a change in the implementation (I will try to find the PR) in the way the nodePort addresses are computed. This is how i should be; On all clusterIP, loadBalancerIP and externalIPs are assigned. So the current implementation is correct. What I suspect that pre v1.23 K8s did this wrong by adding all addresses on interfaces except kube-ipvs0 (including loopback which would not work).

The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without assigning the VIP to any interface. These loadBalancerIP addresses are just the external IP of my nodes, which I use in metallb as an available pool for service(loadbalancer). Provider automatically assigns this address to the physical interface as a lease, I don't really have the option of a different setup at the moment. Ye, I know. In my case the public IP address pool for node and loadBalancerIP addresses are the same
So, the loadBalancerIPs assigned by metallb is actually the real addresses of your K8s nodes?
Yes, these are public IP addresses (each node has its own), which I use as an entry point for some services(loadbalancer) but I also have an external loadbalancer that targets nodePort. To eliminate SPoF, it directs traffic to the service (NodePort), which has been listening on all nodes so far, regardless of IP assignment to another service
Is this setup supported? Thanks for explaining. I understand your setup now, . I am unsure if it's supposed to work, so I have to ask. Have you tested proxy-mode=iptables? It may work, but I wouldn't be suprised if it doesn't.
I will try to test it with iptables, but I use ipvs because of specific services, so I can use the load balancing algorithm - least connection... this method won't solve my problem :(
Found the PR
Why do you need nodePorts services at all? Can't you use only loadBalancer? NodePort is mainly intended to be used by and external load-balancer, i.e. like in AWS, but when you manage your own cluster you usually don't have an external loadBalancer so use only loadBalancerIPs makes sense. We do that, and even .
Scratch that! In your case where the node addresses are the external addresses you should use NodePort only. If you use your node addresses as loadBalancerIP you will hit this PR when you upgrade to later K8s versions.

It inserts a blocking rule in the INPUT chain for loadBalancerIPs. This will block input to your node if loadBalancerIP==nodeIP.
I understand, thanks for the info. In my case it's a breaking change and I will have to migrate some services to NodePort before kubernetes upgrade on prod environment.
with proxy-mode=iptables, everything started working, node is listening on nodePort and I can get to loadBalancerIP, I cleared the rules on each node, just to be sure However this is not a solution for me, I need the lc algorithm from ipvs
Port used in the LoadBakancer service example is within the default NodePort range. I think it can be converted to a NodePort service without affecting external users. Or is just an example? Or are there other reasons for type:LoadBalancer?
/area ipvs
This is just an example, I won't have much trouble with the migration. At most, I will divide the available lease of IP addresses into smaller pools, for NodePort / Loadblancer services. I think I already know everything and there should be no more surprises. Thanks for the information and your time!
It is a little "unusal" to use the host IP as the LB IP, but it's unfortunate that we broke something that used to work. Lars, is there any path back to "working" that doesn't lose the protections in that linked PR? I could easily see iptables also breaking this..
To not exclude loadBalancerIPs (and ecternalIPs) when they are on a physical interface is not hard, but messes up the nice set-operation. The access protection in It should basically not be there if the any loadBalancerIPs (and ecternalIPs) is also on aphysical interface. Can be done, but may be messy.
/triage accepted Even though this issue is accepted, it's still under investigation. I will check the code if the old bevaivor can be restored in a fairly simple way. But there is a problem with back-port. If I make a PR now it will be in v1.26.x or v1.27 but the function was broken in v1.23 so a fix should really be back-ported all the way back.

I would prefer to document that a real address on an interface on the node can't be used as a virtual address, like loadBalancerIP or externalIP.
Just wanted to chime in and say that I've also encountered this issue... very grateful to find the discussion and looking forward to a solution. Thank you all! Also just to share, I've tried the following 3 workarounds and they all work in my case: Switching to iptables mode. It works I think because the iptables NAT rule refers to the hostname as the destination, which gets resolved to the public node IP. Disable IPv6 on my OS. This works too as it makes the code fall back to the single stack mode, which grabs the node IP also from looking up the hostname. Explicitly configure the field in the kube proxy config, that overrides the default get local addresses.
I backed down to K8s v1.22.4 and I can't get it to work unless you have the master outside the cluster (i.e. not a node), or have a setup where the external-ip you set as loadBalancerIP is not on the main K8s network, something like; <img src="" width="65%" /K8s is using the "backend" network, the "external" ips are on the "frontend" network. The reason is when you assign a loadBalancerIP, that address will be assigned to the interface, even before v1.23. So, on an in-cluster master node you will see something like; The address is the loadBalancerIP and is also the node-address of another node in the cluster, "vm-002". When the master receives "normal" packets from "vm-002" they will have src=192.168.1.2 but since that address is assigned to a local interface on the master the packets will be discarded as or spoofed packets. The result is that "vm-002" (e.g. on that node) loses contact with the master;
I discovered this when testing PR I think I can restore the pre v1.23 behavior now, but I suffer from the problem described above.
I don't think your problem is precisely the same. To disable ipv6 or set nodePortAddresses can't really be used as work-arounds for this issue.

Thank you, You're right about IPv6 - disabling it didn't really help. I forgot to remove the I put in while testing with disabling IPv6. It was the that makes the difference. That said, it really appears to work if we explicitly put in the . Maybe I'm still missing something in the code, but if I read it correctly, here is where we take out the load balancer IP: It is called from the folllowing, but only if contains any zero CIDRs. If a node port address is explicitly provided and it is NOT a zero CIDR, then will not be called. We're planning to roll out this workaround before upgrading to a future Kubernetes release with your fix. We would much appreciate it if you could shed more light why it wouldn't work. We didn't want to change to iptables mode if this second workaround really works. Thank you so much!
You are right. To explicitly set in the kube-proxy config works as a work-around. Still the loadBalancerIP (which is also an address on a node) can't be used by the main k8s network, or the master must not be a node in the cluster as described in And... This will work up to and including K8s v1.25.x, but NOT v1.26.0! I haven't checked yet but I think it's because of
Since the loadBalancerIP that is owned by a node is assigned to on other nodes you must stop those other nodes from responding to ARP. For example with:
From my perspective, this is not a big , I have the whole architecture implemented in separate networks. I think it's worth adding this information somewhere in the documentation.
Thank you so much for the response, epesically the info about v1.26. We'll check it out for sure and will report back. Our node port use case is for single-node deployments only, so luckily there are no other nodes.


Question: echo "== Refreshing package database ==" until apt-get update; do echo "== apt-get update failed, retrying ==" <del> echo sleep 5 </del> <ins> sleep 5 </ins> done }

I noticed that there is a huge regression on node bootup since Wednesday last couple of days: Bad one: vs. good one: Please note that the issue is not related to any kubernetes component. The node hasn't run salt client yet. This explains a lot of node not ready issue we have lately. cc/
cc/
Whoa, it shouldn't even be running at all. That implies someone a package recently.
I didn't see any new package is , and the entire package size is the same. :-(
Can you dump /var/log/apt.history (I think it is) into here?
Sorry, . Hmm. I don't see anything new on a recent-ish head node, either, but YMMV.
Yes, just about to do it ... Here you are, and please note that it is from a different node with the same bad symptom:
Yeah, no issue there. Oh. Hm. Sorry, I forgot the one directly in .
Wait. The one in configure-vm is new, it's from , and we need to take it out.
Oh no - so sorry :-( does add an apt-get update to configure-vm. But you are installing bridge-utils from apt, so I thought it was correct to apt-get update first. I can easily remove the apt-get-update and add it back into the AWS version instead. In my experience, crazy-slow apt-get problems are either due to an IPv6 address being served to an IPv4 only host, or getting a bad mirror. Is there a GCE internal mirror? Could also be the gluster repo as well - not necessarily the debian repo.
Probably the bestest thing would be to make sure bridge-utils is on the base image, only apt-get-install bridge-utils if brctl is not available, and run apt-get-update as part of apt-get-install only once per script invocation. But this isn't actually fixing the problem. Something is seriously wrong if an apt-get update takes 12 minutes.
is actually built-in on (the later is a no-op, sorry). I went through an effort at some point to make sure that never needed to or anything, because we were having troubles like this.
I can work up a PR that will probably work for all parties.
Yes, that is the root cause and we want to remove all external dependencies at bootup time. Assign this to
Thanks & sorry again to all.


Question: return true } <del> // Case 2: pod has been assumed and pod updates could be skipped. </del> <ins> // Case 2: pod has been assumed could be skipped. </ins> // An assumed pod can be added again to the scheduling queue if it got an update event // during its previous scheduling cycle but before getting assumed. <del> if sched.skipPodUpdate(pod) { return true </del> <ins> isAssumed, err := sched.SchedulerCache.IsAssumedPod(pod) if err != nil { utilruntime.HandleError(fmt.Errorf("failed to check whether pod %s/%s is assumed: %v", pod.Namespace, pod.Name, err)) return false </ins> } <del> return false </del> <ins> return isAssumed </ins> } func defaultAlgorithmSourceProviderName() *string {

Currently the scheduler may add an assumed pod back to the queue if the update didn't include specific fields (see ). There doesn't seem to be a reason why we can't simply ignore all updates of assumed pods. Assumed pods are waiting to be bound, if binding succeeded then the pod is scheduled. If binding fails, the pod's latest version will be put back into the unschedulable queue anyways. Performance: we save on unnecessary scheduling attempts. This is a followup on the discussion we had in /sig scheduling /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : #### What would you like to be : /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What we basically need to do is: 1) remove function 2) in the places where we call , we should simply call
/assign


Question: # config-default.sh. readonly GCE_MAX_LOCAL_SSD=8 <ins> BASE64_HELP=$(base64 --help) BASE64_INPUT_FLAG="" if [[ $BASE64_HELP =~ "--input" ]]; then BASE64_INPUT_FLAG="--input" else BASE64_INPUT_FLAG="" fi </ins> KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/../.. source "${KUBE_ROOT}/cluster/gce/${KUBE_CONFIG_FILE-"config-default.sh"}" source "${KUBE_ROOT}/cluster/common.sh"

under kubernetes project and running get error: ! I am running on darwin arm64 running get this: if running on linux:
/area test
/sig cli
/sig testing /area test-infra /remove-sig cli /remove-area test
FWIW You pretty much have to install gnu coreutils to develop here, many scripts will actually refuse to run if the gnu version of utils are not available. That said, we should fix this.
/triage accepted


Question: package cm import ( <ins> "errors" </ins> "io/ioutil" "os" "path" "testing" <ins> gomock "github.com/golang/mock/gomock" cadvisorapiv2 "github.com/google/cadvisor/info/v2" utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing" kubefeatures "k8s.io/kubernetes/pkg/features" </ins> "github.com/opencontainers/runc/libcontainer/cgroups" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" <ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" cadvisortest "k8s.io/kubernetes/pkg/kubelet/cadvisor/testing" </ins> "k8s.io/mount-utils" )

When trying to join a second or sometimes third control plane node, fails with: something that looks like an etcd hangup ("Initial timeout of 40s passed") CrashLoopBackOff for one of the kube-apiserver pods The k8s suggest opening an issue here. Successful join op Here goes... First to init the first control plane node (1 of 3 in an HA cluster). This is successful. Contents of : Apply CNI plugin: Looks fine at this point. shows all Running. Next join 2nd control plane node: This seems to go off without a hitch also. The new node comes up as Ready and all new pods Running. Contents of here: Now for the third, failing node: same exact join command, same exact config. It is really a replica node of the 2nd control plane node, and yet times out. Output: Then going back to initial control node. Output of : Uh-oh. So CrashLoopBackOff shows up and one etcd pod is missing completely. No. Kubernetes version (use ): Cloud provider or hardware configuration: AWS EC2 OS (e.g: ): Ubuntu 18.04.5 LTS Kernel (e.g. ): Network plugin and version (if this is a network-related bug): weave-net
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/sig usability
Can you check etcd yaml in node in ? Also check the log on node to know why the etcd pod is not created. A known issue is for ephemeral-storage check, you can remove the ephemeral storage requests in and see if the etcd pod can be created.
Okay, I re-ran this exact setup on a newly provisioned set of nodes and ran into the same error.

xx-xx-xx-248: initial control plane node (ok) xx-xx-xx-131: second control plane node (ok) xx-xx-xx-176: third control plane node (fails to join) Also want to add the comment that I am joining the nodes in serial rather than in parallel, since I have in other issues seen parallel joins described as an issue for etcd. With this latest run I ran with and fetched some additional log info as referenced by Contents of on the failing node: Output from the failed on the failing node: is same as original post. I am -ing the output to and guessing I might need to capture stderr too. But that's outside the scope of this issue. And lastly kubelet log on the failing node (from ): As shown above this is littered wth (newlines for readability): Final note: the IP changes in this syslog from .65 because the nodes are rebooted before any init or join operations are ran.
/sig node storage
Going to give a try. In my situation here, each node is 65G disk and 4G RAM with nothing else really going on on them besides the k8s initialization.
It looks like applying as outlined in provided a band-aid and the final control plane node joined successfully. The param is successfully removed from . For other kubeadm users looking to apply this fix, the for is: The key is that the argument is a directory, not the patch file specifically. In my case, the file sat at and the invocation is then
I've updated PR to reflect that it will fix this issue Also, note that was merged yesterday, and will be cherry-picked into future 1.20 and 1.21 releases.
thanks for checking the logs. let's keep a single tracking issue for this problem: /close
Closing this issue. <detailsIn response to : Jun 09 11:49:50 ip-xx-xx-xx-176 kubelet[4641]: I0609 11:49:50.

4641 ] "Failed to admit pod, unexpected error while attempting to recover from admission failure" pod="kube-system/etcd-ip-xx-xx-xx-176" err="preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: ), ]" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

on the new node fails sometimes. The following error appears in the log: on the new node succeeds every time Join a control plane node The node has over 40 GB of free space on its root partition. I some more logging to the following functions: , , , , , , It appears that the ephemeral storage is detected a bit too late: /sig node storage Kubernetes version (use ): 1.20.4, commit Kernel (e.g. ): 4.9.252-1.ph2
/assign I will try to reproduce it today.
LocalStorageCapacityIsolation is a feature gate for kubelet. Can you check if you disable it? ReadyCondition will check ephemeral-storage as required capacities if this feature gate is enabled in kubelet configuration. Would you check it first?
[x] 1.8 alpha ephemeral-storage [x] 1.10 Promote LocalStorageCapacityIsolation feature to beta [x] 1.15 KEPs Quotas for ephemeral-storage for LocalStorageCapacityIsolationFSQuotaMonitoring default to false is Alpha since 1.15 [x] Failed to promote LocalStorageCapacityIsolationFSQuotaMonitoring feature to beta [x] 1.20 Honor disabled LocalStorageCapacityIsolation in scheduling I am no sure whether there is a plan to GA LocalStorageCapacityIsolation?
I tried to fix it in . Need some testing later.
It appears that the pod fails before sees the storage capacity:
Would you try this ? I build it with on PR branch.
I tried to apply the PR to 1.20.4. After four successful s, one failed. I got the following error:
! It means that during the node resource updated the first time, the cadvisor is not available yet. If we wait here, it would be a problem. I would think more of how to fix it.
Hi, Same problem here. I'm using K8s v1.20.4 with CentOS7 (2 cores, 4GB and 20GB storage) Cluster creation with kubeadm init always works, but when I try to add several master nodes with kubeadm join it takes a lot of time waiting for etcd pod to be created. In most of cases, etcd pod never runs.

The rest of components (APIserver, controller, scheduler and proxy) exists and works (except APIServer which reboots trying to connect the missing etcd) Checking the kubelet logs in the missing etcd hosts I have found this: Mar 10 14:23:27 kmaster-02 kubelet[9448]: W0310 14:23:27. 9448 ] Failed to admit pod etcd-kmaster-02_kube-system() - Unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: ), ] That node have 17GB storage free. If kubelet is restarted, the etcd (pod and container in docker ps -a) appears. If I could help doing something please tell me. I would like to help
For anyone who is experiencing this issue and still wants to use kubeadm until it is fixed: You can remove the ephemeral-storage request from the etcd static pod and the join will work. Just make sure that your nodes actually have more than 100MB free storage left. You can reinsert the storage request by patching the etcd manifests after the nodes are up. To patch etcd during a create the following file (the filename matters, the file location doesn't) and use to join additional control plane nodes.
Thanks for the workaround I'm encountering this same issue and the workaround worked for me. Edit: the patch I tried doesn't work though (not sure if this is really a problem). I issued the command: and the response shows that the resource can't be patched this way: There may be another way but I'm still fairly new to Kubernetes so it is not obvious to me at the moment. Anyhow, thanks for the suggestion!
I updated Add a start-delay of 10 seconds for a condition that the capacity of node ephemeral-storage is 0. Not sure if this will fix it.
we are starting to see reports about this every week.
cc
cc
From it sounds like the issue is kubeadm etcd ephemeral-storage requests are too big for some environments? So far this is sounding like a kubeadm issue, not a kubernetes issue.
it seems to be something else.

i doubt that users are trying machines that don't satisfy 100Mi. for example, the report in the OP points out that the machine has 40GB.
I can confirm that the VMs on which I installed k8s control plane components had sufficient disk space when I encountered the problem. I'm running Kubernetes v1.20.4 and kubeadm v1.20.4.
I think the root cause here sounds right. There is kind of race here. adding delay will help, but it does not have 100% guarantee. Not sure whether it is easy to add logic to make sure capacity is reported correctly before starting pods..
Did something change in 1.20 regarding the ordering of cadvisor and node startup?
If the reason would be a disk problem(or something we can figure out) and the delay can solve it, I think we should add a warning besides the delay.
We need to figure out what triggers the race condition. This only happens on some machines. I have not seen this myself.
We're able to repro this nearly 100% of the time with Cluster API Provider Azure + k8s 1.20.x + Ubuntu 18.04 + default template, which makes k8s 1.20 pretty much unusable at this point. Trying to see if this repros with other non-Azure cluster-api providers with kubeadm 1.20.
Does anyone know why this only repros on and not ? The ephemeral storage resource requirement here is used in common by both and : In practice I can workaround this issue in cluster-api + Azure by adding a sleep command to my spec, but it requires a significant number to make work ~100% of the time (like 90 seconds). Does the flow take significantly longer before it gets to the part where it submits an etcd pod to the apiserver? Is there any other explanation for this being a -only symptom?
i can see this failing for any static pod with ephemeral storage requests. that is, unless there is proof that this is somehow specific to the kubeadm static pod for etcd that kubeadm deploys.
I agree that the specific kubeadm-delivered etcd pod is not interesting. (In fact it's very permissive, it's only asking for 100Mi.)

What definitely seems to be happening (in Azure) is that kubelet takes a long time to know about its underlying node's ephemeral-storage availability. Kubelet starts: ~2 mins later, it complains that it can't schedule etcd: Again, why does this not repro on nodes created by ?
it also seems to be an even more scoped problem because it's not reproducible for me locally (i create a number of clusters every day, usually), the kubeadm CI or in other high level projects that use kubeadm (e.g. kind, minikube, etc). we should try to find a common reason / condition from all the reporting users that are seeing this.
Experiencing same problem on openSuSE Kubic/Tumbletweed snapshot with kubeadm/kubelet 1.20.2, kernel 5.11.15 running on Xen, 2 CPU, 4GB ram 30GB disk (BRTFS)
Problem consistently appear/disappear when switching between snapshots (Working) and , so a package update in that period has probably triggered this. Following files contains changes between those two snapshots.
are you suggesting a root cause in underlying Linux storage driver surface area that is adding delay/uncertainty to kubelet's ability to introspect available ephemeral-storage on the node host OS?
A bit of luck combined with 6th sense i got it narrowed down to systemd update here Looking at the changelog, one item sticks out: Anyone else enabled cgroupsv2 on i.e. the CentOS cases (it needs to be enabled explicitly for that AFAIK)?
ACK on your latest respond regarding unified cgroup hierarchy. Is there a quick, cross-distro way to verify that any folks who are repro'ing have a similar systemd config?
The precence of file/dir /sys/fs/cgroup/cgroup.controllers should mean you are using cgroupsv2 ( Ref: ) I can confirm that the file is present on the kubic snapshot which is not working. It is not present on the one that does work. ( I'm using cri-o as runtime which should support cgroupsv2 ) cgroup2 support was introduced for K8S in 1.19 ( ).
Thanks. So tl;dr your working theory is that cgroupsv2 + kubelet ephemeral-storage admission has some pathologies.

I'll post quickly with what my repro containerd + Ubuntu 18.04-LTS OS surface area looks like.
I ran into this issue consistently on Ubuntu 20.04.2 LTS using cri-o but is not present on my systems.
I can also confirm that cgroupsv2 is not a factor in my repro'ing environment:
Also, I'm not using cri-o, containerd here:
Ok - so much for that theory. But still: What I did to narrow it to systemd-update was taking the working kubic snapshot () and run zypper update systemd which updates systemd (along with systemd-sysvinit and udev packages to same version). That consistently made it fail. So I think a pointer to the root-cause might still be hiding somewhere around this changelog - even if it is not cgroupsv2. That part was just a theory (seemed like a "big" thing), but the update of systemd package is fairly hard evidence.
I have identified where these particular symptoms (probably) originate, and cursory testing suggests something like this eliminates the issue: I'll be advocating for a few mins in today's sig-node discussion if any folks following this thread want to chime in. cc
/sig cluster-lifecycle /triage accepted
I've also observed that on some occasions the node allocatable storage is in a continual state of reporting "0". Restarting kubelet immediately fixed that. More debugging to do....
did you manage to reproduce this with an arbitrary static pod that is not the etcd one?
I was not able to reproduce in a non-kubeadm test, but I suspect that the way kubelet bootstrap is implemented there (and possibly environmental reasons) the static pod handling was always happening after this code executed: (the above is in the containerManager method) kubeadm w/ ephemeral-storage requirements in etcd is the most effective repro at this time
the writing of the etcd pod manifest in during is after the kubelet has already started and i wonder if that is contributing to the observed failures. probably not given this is a cadvisor related problem. to verify that i would write an arbitrary static pod manifest with the same request in : before (which also starts the kubelet) right after the kubelet has started if is causing such racy problems i don't think it should be graduated to GA.

though, it has been Beta for ~12 releases (on by default), so it's already used everywhere.
Based on your assessment do you think we should remove the ephemeral-storage requirement from default kubeadm etcd static pod? I can make that change.
based on the investigations here (thanks all), it feels like is a feature that may or may not work (for static pods at least). it doesn't feel right to mitigate a kubelet bug on the side of the deployer, but given how many reports there are about this i think it should be done at this point. happy to LGTM approve your PRs. note it must also be backported to all versions in support (back to 1.19). suggesting the following release note: we should keep this issue open, too.
Sounds good. I'll submit a PR soon with the kubeadm change, and I'll continue to drive toward a root cause solution on the kubelet side. I'll bring this up at sig-node tomorrow. Thanks!
is a possible duplicate of this issue - seeing a CrashLoopBackOff failure for kube-apiserver under a similar scenario to those mentioned here, e.g. failure to join a third control plane node with mentions of in the kubelet logs.
An ugly as sin workaroud: (please don't hate me :D )
It seems like this could be closed with and related cherry picks now being part of the newest releases? Bumping to 1.21.2 allowed me to remove the hack for etcd and several test deployments have steered clear entirely of the original issue. Thanks
great news. I'd prefer to pin this issue to this PR: That PR will deliver more race condition resilience for all static pod + ephemeral-storage requirement scenarios.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:


Question: v1helper "k8s.io/kubernetes/pkg/api/v1/helper" "k8s.io/kubernetes/pkg/cloudprovider" "k8s.io/kubernetes/pkg/controller" <ins> kubefeatures "k8s.io/kubernetes/pkg/features" </ins> "k8s.io/kubernetes/pkg/util/metrics" )

For various reasons (network topology, heterogenous nodes, etc) a user may want to control the addition of a node to a cloud load balancer. A specific issue with the Azure Container Instances connector which introduces a virtual node can be found here: The right way to do this is via a node annotation, e.g.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen This is still a problem.. any word on when feature gate goes out of alpha?
Setup the enhancement tracking issue for this Still discussing with sig-network on how we want to move forward with this.


Question: nginx "$@" oldcksum=`cksum /etc/nginx/conf.d/default.conf` <del> inotifywait -mr --timefmt '%d/%m/%y %H:%M' --format '%T'  </del> <ins> inotifywait -e modify,move,create,delete -mr --timefmt '%d/%m/%y %H:%M' --format '%T'  </ins> /etc/nginx/conf.d/ | while read date time; do newcksum=`cksum /etc/nginx/conf.d/default.conf`

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT on sample file I was using the /examples/https-nginx sample application for some small testing purposes and was surprised by the relative high load it kept having in the nginx pods. After some inspection I found out that kubernetes/examples/https-nginx/auto-reload- is basically running a busy loop. the command doesn't filter out some of the events it should filter out. At least on my system with just this sample config I always get a OPEN on basically immediately so I would recommend adding as arguments to the inotifywait command
I can take this.
I already have a PR in flight for this.
OK thanks for info


Question: }, }, Sync:       false, <del> PollPeriod: time.Second * 20, </del> <ins> PollPeriod: time.Second * 2, </ins> Timeout:    time.Second * 20, } }

I'd like to make the case for moving the check out of the Storage interface and put that responsibility at a higher level. It was introduced . The case for moving it up 1) the interface it's implemented under already provides the primitives necessary to reproduce the check (GetPod) 2) it couples creation storage tightly to the (unreliable) infrastructure, which means consumers of the interface must anticipate both apiserver, storage, and infrastructure issues (vs just apiserver and storage) 3) as an API consumer I may want to know when my pod moves to pending, vs to started (which could be multiple orders of mangnitude longer) Would there be strong disagreement if I opened a pull moving the async wait to above the interface? Or is there a strong design reason to couple this here that I missed?
Hey Clayton, I think in general, I was trying to have it only return when it was "done" but as you point out, there are lots of problems with this, I think that falling back to "in our control" is perhaps the correct semantic. We've been thinking along similar lines, namely that we should only do acceptance testing (e.g. "is this pod feasible") before returning 200 and stopping waiting, instead of waiting for the task to go fully running. I took a rough design that proposed, and pasted it into: So if that design makes sense to you, and achieves the the same goals as what you are suggesting (I think it does) let's move towards that. The first thing might to roll back the other PR, and then move forward from the previous state. Thanks --brendan
Agree that we're "done" when the pod is accepted and stored in the system, and needn't wait until it's running. Ultimately, one should be able to wait on events (i.e., we'd wrap etcd's watch); one such event would be the state change from assigned to running, and that's how clients should get this behavior if they need it.
Agreed
This is fixed


Question: registry, prof := initRegistryAndConfig(permitPlugin) // Create the master and the scheduler with the test plugin set. <ins> // TODO Make the subtests not share scheduler instances. </ins> testCtx := initTestSchedulerForFrameworkTest(t, testutils.InitTestMaster(t, "permit-plugin", nil), 2, scheduler.WithProfiles(prof), scheduler.WithFrameworkOutOfTreeRegistry(registry))

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestCoSchedulingWithPermitPlugin Testgrid link: Reason for failure: /sig scheduling
Could be related
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Another run
Hmmm I think test plugin is no longer correct: After my change is executed at the end of the scheduling cycle, this means there can be only one happening at any given moment. If there is a race where "signaling-pod" is scheduled before "waiting-pod", then "signaling-pod" will wait 30s in in for the "waiting-pod" to become waiting, but this cannot happen. Basically "signaling-pod" is allowing "waiting-pod" before "waiting-pod" is being scheduled. I think test and test plugin could be changed, so there are no longer two pods of different kind. Instead first pod that arrives to the is considered "waiting-pod" and second pod is always "signaling-pod" that will either reject or allow first pod. This way we do not have to care about order in which scheduler will process these pods. I think it is also consistent with how actual co-scheduling will work.
Can you first confirm that this is what is happening? you could increase the log level to 3 to see the log lines in and . I feel like if that's the case, we have a separate bug with the event handlers or sorting. Other than that, I agree that the implementation of the test's shouldn't have waits. cc
My hope would be that the pod is temporarily unschedulable due to nodes still being synced.
Where you able to confirm the reason of the flake?
not yet, what is the easiest way to run this test multiple times? Should I try to reproduce this locally or create some temporary PR just for this purpose?

I wrote PR for the test fix to remove sleep from the PermitPlugin, it had to be done sooner than later regardless of the cause of flakiness.
Understanding the reason of the flake is important to make sure we are fixing the right thing and that we won't have future flakes. Please try to run the tests locally about 10 times.
It should be something like: If that doesn't work, run first
Flake point:
From the logs provided by you can tell that the problem is a race between "signalling-pod" and "waiting-pod" and waiting inside the Permit(), as I mentioned in this issue. BLUF: In the first case of the TestCoSchedulingWithPermitPlugin "waiting-pod" and "signalling-pod" pods are being retried many times, most of the time "signalling-pod" is rejecting already waiting "waiting-pod", but sometimes race can happen and "signalling-pod" will be processed by the scheduler before rejected "waiting-pod" (there is no guarantees whether rejecting pod or rejected pod will end up in the queue first). When this happens "signalling-pod" waits 30s inside the Permit() for "waiting-pod" to become waiting and blocks the scheduler, meanwhile second test case starts, "signalling-pod" eventually stops waiting for "waiting-pod" and gets bound, after that "waiting-pod" cannot be allowed, which leads to a failure of the second test case. My PR should fix this issue.
I feel like we should avoid sharing the scheduler & permitPlugin instances among the 2 subtests, which can then solve the flake?
I think this flake (in different variations) can happen as long as test depends on the order in which signalling and waiting pod enter the permit plugin.
I believe the PR might fix the issue. But since the reason for the flake is that one subtest is corrupting the other, it's possible that we might surface other flakes. That's why it's important that you run the test several times to check for other flakes. It several tests pass, let's proceed with this PR and leave a TODO to remove the dependency between the subtests.
I ran 10+ times on my branch with the fix and had no issues.


Question: if len(namespace) == 0 { namespace = metav1.NamespaceDefault } <del> selfLink = fmt.Sprintf("/api/"+api.Registry.GroupOrDie(api.GroupName).GroupVersion.Version+"/pods/namespaces/%s/%s", name, namespace) </del> <ins> selfLink = fmt.Sprintf("/api/"+api.Registry.GroupOrDie(api.GroupName).GroupVersion.Version+"/namespaces/%s/pods/%s", namespace, name) </ins> return selfLink }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): selflink kubelet selflink default selflink manifest selflink Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): admin kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-27T10:00:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-27T09:42:05Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: AWS OS: Debian GNU/Linux 8 (jessie) Kernel: Linux ip-10-150-109-115 4.4.41-k8s SMP Mon Jan 9 15:34:39 UTC 2017 x86_64 GNU/Linux Install tools: kops 1.5.3 Others: What happened: When running the selfLink for pods started from manifests were incorrect. Example: What you expected to happen: To receive a correct selfLink: Example: How to reproduce it (as minimally and precisely as possible): I believe simply using a pod manifest to start a pod will cause this issue as kubelet seems to rely on defaults when starting pods from manifests. Anything else we need to know: I believe the below linked line needs to be corrected:
/assign I reproduced it. Will fix. Thanks for reporting!
GitHub didn't allow me to assign the following users: xiangpengzhao. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details


Question: :KUBE-NODEPORTS - [0:0] :KUBE-POSTROUTING - [0:0] :KUBE-MARK-MASQ - [0:0] <del> :KUBE-MARK-DROP - [0:0] </del> :KUBE-SVC-AHZNAGK3SCETOS2T - [0:0] :KUBE-SEP-PXD6POUVGD2I37UY - [0:0] :KUBE-SEP-SOKZUIT7SCEVIP33 - [0:0]

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-cos-master-alpha-features Which test(s) are failing: [sig-network] Networking should recreate its iptables rules if they are deleted [Disruptive] Since when has it been failing: as far as testgrid can see, since forever, it's like ... 10% flake Testgrid link: Reason for failure: Anything else we need to know: /cc /sig network /priority important-soon /milestone v1.17
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
FTR, I'm pretty sure that the feature is working fine and it's only the test that is broken. The kubelet logs (eg, ) show that it is noticing the deleted chains and reloading: I1119 10:56:23. 1325 ] iptables canary mangle/KUBE-KUBELET-CANARY deleted I1119 10:56:23. 1325 ] Reloading after iptables flush And if it failed to recreate any of the rules, it would log errors about it: if _, err := kl.iptClient.EnsureRule(utiliptables.Append, utiliptables.TableNAT, KubeMarkDropChain, "-j", "MARK", "--set-xmark", dropMark); err != nil { klog.Errorf("Failed to ensure marking rule for %v: %v", KubeMarkDropChain, err) return } and we don't see those errors in the logs. So I think there's just a bug in the test case. Trying to reproduce now in a debug PR...
my apologies, this is the cause of the flakiness The test tries to find the KUBE-MARK-DROP rule but kube-proxy deletes the rule because we are not recreating it as we are doing with the KUBE-MARK-MASK Since is only needed for kube-proxy with dual stack in iptables mode and that PR is not merged we can revert it. The other option is making the the chains in kubelet and kube-proxy different


Question: // no configuration is an error, some parameters are required if len(data) == 0 { <del> return nil, fmt.Errorf(errfmt, fmt.Errorf("config file was empty, but some parameters are required")) </del> <ins> return nil, fmt.Errorf("init config file %q was empty, but some parameters are required", path) </ins> } return utilcodec.DecodeKubeletConfiguration(loader.kubeletCodecs, data)

From in The most important things I want to see through node e2e tests: 1) the kubelet is started from last-known-good-config file 2) kubelet can download the new config properly to the node; and restart itself from the new config file 3) kubelet can mark the config bad properly after several round of crashing caused the bad config 4) Kubelet can roll back the last-known-good-config and restart itself from that good one 5) kubelet can properly reports its configuration status Note: (2) is covered by existing e2enode tests (1, 4, 5) were covered by the , all of which I plan to turn into automated tests after this PR merges - but I'm trying not to increase the size of this PR any further ~(3) we need to figure out how to test this. I don't want to rely on a "valid" config that causes a crash loop - because the very existence of such a config should be viewed as a bug. I also don't want to include a "please crash loop for our tests" field in the config, because that's an ugly API. The best solution is not clear here.~ We decided to remove crash-loop detection, see . Filing this issue to track this work: into e2e_node tests [x] remote config [x] local (from file) config [x] ~test the crash-loop detection~ removed, see above /cc
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.8 milestone for more than 7 days: kind: Must specify exactly one of [, , ]. priority: Must specify exactly one of [, , ]. Removing it from the milestone. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
The test cases in the OP have been covered by our e2e_node tests for a while.


Question: internalELB := false internalAnnotation := apiService.Annotations[ServiceAnnotationLoadBalancerInternal] if internalAnnotation != "" { <del> if internalAnnotation != "0.0.0.0/0" { return nil, fmt.Errorf("annotation %q=%q detected, but the only value supported currently is 0.0.0.0/0", ServiceAnnotationLoadBalancerInternal, internalAnnotation) } if !service.IsAllowAll(sourceRanges) { // TODO: Unify the two annotations return nil, fmt.Errorf("source-range annotation cannot be combined with the internal-elb annotation") } </del> internalELB = true }

Kubernetes version (use ): v1.4.6 Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS v1235.2.0 Kernel (e.g. ): 6-coreos-r1 SMP Wed Dec 7 00:29:42 UTC 2016 x86_64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz GenuineIntel GNU/Linux What happened: When creating a service-type loadBalancer, adding creates an internal ELB, but the only allowed value for is 0.0.0.0/0. This limitation prevents you from specifying custom CIDR range for your hybrid cloud (Direct Connect to private data center) What you expected to happen: The limitation is unnecessary. Even in an AWS only scenario (no hybrid cloud), it makes sense to allow a CIDR range for a subnet where your calling services are deployed. How to reproduce it (as minimally and precisely as possible): Create a service-type loadBalancer and set loadBalancerSourceRanges to a value other than 0.0.0.0/0
Any update on this? Also, is anyone working on adding support for securityGroup?
any progress on this? I also really would like to have this flexibility in place for integrating k8s into an existing VPC.


Question: // This error is ok, beacuse we will delete the pod before it completes initialization framework.Logf("error from create uninitialized namespace: %v", err) }() <del> podB = waitForPodInNamespace(f.ClientSet, podB.Namespace, podB.Name) </del> <ins> podB = waitForPodInNamespace(f.ClientSet, namespace.Name, podB.Name) </ins> By("Deleting the namespace") err = f.ClientSet.Core().Namespaces().Delete(namespace.Name, nil)

Test is constantly failing. https://k8s- https://k8s- /cc
After was manually merged last Friday, serial tests is back to normal flaky stage except this test which is constantly failing. Raise the priority.
i will take a look.
Is supposed to fix this?
they seem like different issues. is about slow namespace deletion. This seems to be an issue with the namespace objects being created?
spotted a bug in the tests. Sent
-- i think this is a separate issue.. for some reason, an empty namespace name is being passed into client which doesnt appear possible at first glance.
I'm going to necro this bug because I figured out why it's failing pod starts trying to get pod namespace starts trying to get the service account secret, but it's been deleted keeps trying and trying and trying to mount, but goes into backoff ~80s later it finally gives up and unmounts the secret it never mounted Logs above are from 1.12. When a pod is detected as deleted, we're trying to unmount a secret that can't / doesn't exist. The fact that the pod is deleted should abort mount much faster, even if mount is in backoff. this is a flaky conformance test (and fixing this should have an impact on other failure modes like this).
It is different than the other cause, so probably deserves a second issue, but we should assess which linked issues are actually this problem and link those into the final fix.
cc fixed a secret teardown issue in the past
/area conformance
I am not sure if it is a bug as such. It is true that when pod is removed, volumemanager tries to mount the volume anyways and since it fails to fetch secret, it keeps failing. to fetch secret is a non-fatal error and volumemamanger is right in retrying that operation. think bit that is problematic is - mount and unmount operation basically share the exponential backoff . I have filed to track this. A workaround might be to increase the wait period in following code: But I can see how having this shared exp. backoff may have domino effect on other tests and may be pod teardown time can be improved in general if mount and unmount did not share the exp. backoff.

/assign
As far as I can see the issue has now been resolved; on test grid I see only green for this test: https://k8s- https://k8s- was the PR that seems to have fixed this. /close
Closing this issue. <detailsIn response to : As far as I can see the issue has now been resolved; on test grid I see only green for this test: https://k8s- https://k8s- Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ${GKE_REQUIRED_SKIP_TESTS[@]:+${GKE_REQUIRED_SKIP_TESTS[@]}}  ${GCE_DEFAULT_SKIP_TESTS[@]:+${GCE_DEFAULT_SKIP_TESTS[@]}}  ${GCE_FLAKY_TESTS[@]:+${GCE_FLAKY_TESTS[@]}}  <ins> ${GCE_SLOW_TESTS[@]:+${GCE_SLOW_TESTS[@]}}  </ins> )"} ;;

For example: I'm disabling the test in But we need to re-enable it, or roll-back the test.
fyi
Shouldn't this is part slow test, and the fix should be either: 1) Increase Kubernetes-e2-gke-ci timeout value if decide to include slow to that project or 2) disable all slow tests on that project? cc/ Which one is required here?
The test runs 50 pods per node, which is over the default --max-pod limit of kubelet. The test relies on the flag --max-pods=100 being set in cluster/gce/config- I will disable it for all GKE tests in On the other hand, it's not clear to me where the timeout is set for a specific build. /cc
Timeouts for each Jenkins build are in the Jenkins project configurations themselves, which are not currently saved in version control anywhere. :
Discussed with about the test offline. There are several issues: Do we want to run any kind of kubelet resource monitoring test on GKE? Is it useful? We only override max-pods in GCE. Should the test itself figure out what platform it can run based on the #pods in the test (see ) Per-test configuration allows greater flexibility (e.g., we may run 100 pods on GCE to test kubelet's scalability), but it'd make the tests more complicated as they need to be aware of the test environment (which many e2e tests already do); there is also the burden of updating the tests once the max-pod value changes. Does this test belong to SLOW tests or should it belong to a new category? Not all slow tests have the same attributes. Arguably this test could be enabled only on selective builds. The downside would be more flags -- less readable/manageable e2e configs. Not all builds which run SLOW tests have the proper timeout value to run this test. We need to manually change them every time a slow test is . Re-assigning this back to since she has more concerns about the PR.
Had offline discussion with We are going to split current Kubelet resource usage tracking test into two, one is targeted for GCE, another one is for GKE. In this case, we won't mess up the test code itself by introducing knobs. Re-assign this back to


Question: // NodeMetrics sets resource usage metrics of a node. message NodeMetrics { <ins> // Standard object's metadata. // More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata // +optional </ins> optional k8s.io.apimachinery.pkg.apis.meta.v1.ObjectMeta metadata = 1; // The following fields define time interval from which metrics were

no explanation for metadata filed of NodeMetrics/PodMetrics, code: should be like $ kubectl explain NodeMetrics KIND: NodeMetrics VERSION: DESCRIPTION: NodeMetrics sets resource usage metrics of a node. FIELDS: apiVersion <stringAPIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: kind <stringKind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: metadata <Objecttimestamp <string-required- The following fields define time interval from which metrics were collected from the interval [Timestamp-Window, Timestamp]. usage <map[string]string-required- The memory usage is the memory working set. window <string-required- and $ kubectl explain PodMetrics KIND: PodMetrics VERSION: DESCRIPTION: PodMetrics sets resource usage metrics of a pod. FIELDS: apiVersion <stringAPIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: containers <[]Object-required- Metrics for all containers are collected within the same time window. kind <stringKind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: metadata <Objecttimestamp <string-required- The following fields define time interval from which metrics were collected from the interval [Timestamp-Window, Timestamp]. window <string-required- able to see the explanation for metadata filed of NodeMetrics/PodMetrics Kubernetes Version: v1.22.0-rc.0+
/sig instrumentation
/kind documentation /assign /triage accepted


Question: return } } <del> Failf("Timed out after %d seconds waiting for %s pods to reach valid state", podStartTimeout.Seconds(), updateDemoSelector) </del> <ins> Failf("Timed out after %v seconds waiting for %s pods to reach valid state", podStartTimeout.Seconds(), updateDemoSelector) </ins> } type updateDemoData struct {

Implement guestbook test as a kubectl-driven test in . this should be a fairly straightforward follow up from , if you are interested/have cycles.
As I wrote in the email the plan is: problem w guestbook example (). test becomes actual test (). it do ginkgo. Should be done this week, but it depends on how fast 1. can be done.