Question: // Only start the http server if port > 0 if port > 0 { serv := &http.Server{ <del> Addr:    fmt.Sprintf(":%d", port), </del> <ins> Addr:    net.JoinHostPort(address, strconv.Itoa(int(port))), </ins> Handler: mux, }

cAdvisor provides potentially sensitive data and there's currently no way to block access with anything other than iptables.
What type of cluster are you running (GKE, or what instructions did you use to setup your cluster?) A short term solution might be to try adding flag to kubelet and restarting the kubelet service and see if the sensitive information is no longer leaked. Doing this may the UI and heapster which expect to be able to poll the cadvisor port. I think probably knows if there is a longer term solution planned, such as: only serve on specified interface. if this is localhost, then it breaks the ui, but maybe that is okay. maybe has thought about this. require authentication/authorization to connect to cAdvisor port.
Self deployed cluster, the kubelet is started via
We are working on improving the kubelet APIs which will subsume cadvisor APIs. Until then setting cadvisor port to '0' is probably the way to go.
Is there an issue and target date for doing this?
Filed . The current plan is to complete this for v1.1. cc
Do you mean the read-only kubelet port ( by default IIRC)? If so, then we really need to disable this & provide authorisation options on the kubelet to be able to retrieve stats fro the kubelet via the normal port with a restricted token/cert/auth options. We've discussed this for OpenShift (openshift/origin) too.
Sorry I didn't realise that cadvisor was bound to port 4194 as well. Thought this was all embedded inside the kubelet process without exposing any ports.
Any reason why we start up cadvisor http server at all? Seems unnecessary now kubernetes embeds cadvisor & exposes via & ?
does not expose all the containers (, , etc). is working on introducing a better stats API in kubelet () that should help us shut down cadvisor's http server in kubelet.
Couldn't we just add cadvisor as a handler in the kubelet anyway? No need for separate http server & would mean access to raw data is available if anyone wants it.
Indeed that is an option. But we need a better stats API in kubelet anyways for reasons mentioned in .
Absolutely agree but thinking quick win.
+1 on a quick win.

We just disabled it by default instead: Which reminds me it would be great to have so we could make it easy to turn on/off stuff like this w/o rewriting the entire command-line in a service file.
cc
it's not 100% clear to me whether or not made it safe to use , could you please clarify? On a fresh k8s 1.6.4 cluster still exposes cAdvisor to the world: the file you referenced was deleted as part of coreos/coreos-overlay the documentation @ (CoreOS docs) doesn't mention cAdvisor should be disabled either. Another reason for my question to Vish.
Ahh. That's a bug that the new component config based scheme introduced. being is a valid setting.
I think we can change this. The KubeletConfiguration type is still technically alpha, so it should be ok to make this field a pointer and keep the current default when it's not provided (but allow 0). This won't affect anyone who presently omits the flag and expects port 4194. Theoretically people could be explicitly passing 0 and expecting 4194... I guess... but that's sort of an insane case, and I'd consider that behavior a bug.
Thanks for clarifying guys :+1: The original problem remains though: enabling the cAdvisor port makes the kubelet listen on all interfaces.
Yeah. We can get cAdvisor to bind to the interface that kubelet uses. cc , Antoine Cotten <:
/assign


Question: } } <ins> func testFinalizerValidationApplyCreateAndUpdateCRD(t *testing.T, rest rest.Interface, gvk schema.GroupVersionKind, gvr schema.GroupVersionResource) { var testcases = []struct { name                 string finalizer            []string updatedFinalizer     []string opts                 metav1.PatchOptions expectUpdateWarnings []string expectCreateWarnings []string }{ { name:      "create-crd-with-invalid-finalizer", finalizer: []string{"invalid-finalizer"}, expectCreateWarnings: []string{ `metadata.finalizers: "invalid-finalizer": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers`, }, }, { name:      "create-crd-with-valid-finalizer", finalizer: []string{"kubernetes.io/valid-finalizer"}, }, { name:             "update-crd-with-invalid-finalizer", finalizer:        []string{"invalid-finalizer"}, updatedFinalizer: []string{"another-invalid-finalizer"}, expectCreateWarnings: []string{ `metadata.finalizers: "invalid-finalizer": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers`, }, expectUpdateWarnings: []string{ `metadata.finalizers: "another-invalid-finalizer": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers`, }, }, { name:             "update-crd-with-valid-finalizer", finalizer:        []string{"kubernetes.io/valid-finalizer"}, updatedFinalizer: []string{"kubernetes.io/another-valid-finalizer"}, }, { name:             "update-crd-with-valid-finalizer-leaving-an-existing-invalid-finalizer", finalizer:        []string{"invalid-finalizer"}, updatedFinalizer: []string{"kubernetes.io/another-valid-finalizer"}, expectCreateWarnings: []string{ `metadata.finalizers: "invalid-finalizer": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers`, }, }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { kind := gvk.Kind apiVersion := gvk.Group + "/" + gvk.Version // create the CR as specified by the test case name := fmt.Sprintf("apply-create-crd-%s", tc.name) finalizerVal, _ := json.Marshal(tc.finalizer) applyCreateBody := []byte(fmt.Sprintf(crdApplyFinalizerBody, apiVersion, kind, name, finalizerVal)) req := rest.Patch(types.ApplyPatchType). AbsPath("/apis", gvr.Group, gvr.Version, gvr.Resource). Name(name). Param("fieldManager", "apply_test"). VersionedParams(&tc.opts, metav1.ParameterCodec) result := req.Body(applyCreateBody).Do(context.TODO()) if result.Error() != nil { t.Fatalf("unexpected error: %v", result.Error()) } if len(result.Warnings()) != len(tc.expectCreateWarnings) { for _, r := range result.Warnings() { t.Logf("received warning: %v", r) } t.Fatalf("unexpected number of warnings, expected: %d, got: %d", len(tc.expectCreateWarnings), len(result.Warnings())) } for i, expectedWarning := range tc.expectCreateWarnings { if expectedWarning != result.Warnings()[i].Text { t.Fatalf("expected warning: %s, got warning: %s", expectedWarning, result.Warnings()[i].Text) } } if len(tc.updatedFinalizer) != 0 { finalizerVal, _ := json.Marshal(tc.updatedFinalizer) applyUpdateBody := []byte(fmt.Sprintf(crdApplyFinalizerBody, apiVersion, kind, name, finalizerVal)) updateReq := rest.Patch(types.ApplyPatchType). AbsPath("/apis", gvr.Group, gvr.Version, gvr.Resource). Name(name). Param("fieldManager", "apply_test"). VersionedParams(&tc.opts, metav1.ParameterCodec) result = updateReq.Body(applyUpdateBody).Do(context.TODO()) if result.Error() != nil { t.Fatalf("unexpected error: %v", result.Error()) } if len(result.Warnings()) != len(tc.expectUpdateWarnings) { t.Fatalf("unexpected number of warnings, expected: %d, got: %d", len(tc.expectUpdateWarnings), len(result.Warnings())) } for i, expectedWarning := range tc.expectUpdateWarnings { if expectedWarning != result.Warnings()[i].Text { t.Fatalf("expected warning: %s, got warning: %s", expectedWarning, result.Warnings()[i].Text) } } } }) } } </ins> func setupCRD(t testing.TB, config *rest.Config, apiGroup string, schemaless bool) *apiextensionsv1.CustomResourceDefinition { apiExtensionClient, err := apiextensionsclient.NewForConfig(config) if err != nil {

A client is allowed to add custom finalizers to Custom Resources with a finalizer name that's not fully qualified (e.g. or ). However they're prevented from doing the same on builtin resource types, returning the following error: Invalid value: "": name is neither a standard finalizer name nor is it fully qualified Per , custom finalizer names are expected to be in format "consist of a domain name, a forward slash and the name of the finalizer" (e.g. ). Therefore, I expected the enforcement to be the same for CRs and builtin types. a custom resource on the cluster it to add the edit does not fail the same on a builtin resource (e.g. Namespace or Pod), observe it fails No response Reproducible at least starting from 1.21 and still on latest versions. <details</details<detailsN/A kind cluster </details<detailsN/A - kind cluster accessed via macOS </details<details</details<details</details<details</details
/area api /cc /sig api-machinery
/assign /triage accepted
There is no validation of finalizer in and . Please unassign me if you guys have started to work on it. /assign
it is called from here on custom resources:
lacks validation for finalizer name. I have raised a pr for this. PTAL.


Question: Service: &apiregistrationv1.ServiceReference{ Namespace: namespace, Name:      "sample-api", <del> Port:      pointer.Int32Ptr(aggregatorServicePort), </del> <ins> Port:      pointer.Int32(aggregatorServicePort), </ins> }, Group:                "wardle.example.com", Version:              "v1alpha1",

[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below.

This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted


Question: */ framework.ConformanceIt("should function for intra-pod communication: udp [NodeConformance]", func() { config := e2enetwork.NewCoreNetworkingTestConfig(f, false) <del> for _, endpointPod := range config.EndpointPods { config.DialFromTestContainer("udp", endpointPod.Status.PodIP, e2enetwork.EndpointUDPPort, config.MaxTries, 0, sets.NewString(endpointPod.Name)) } </del> <ins> checkNodeConnectivity(config, "udp", e2enetwork.EndpointUDPPort) </ins> }) /*

What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: Namespace:       ns, ResourceVersion: "1", Labels: map[string]string{ <ins> "a":                  "b", </ins> v1.IsHeadlessService: "", }, },

bug (): Originally posted by in cc /sig network /priority critical-urgent
fix in
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: if failSwapOn { // Check whether swap is enabled. The Kubelet does not support running with swap enabled. <del> swapData, err := ioutil.ReadFile("/proc/swaps") </del> <ins> swapFile := "/proc/swaps" swapData, err := ioutil.ReadFile(swapFile) </ins> if err != nil { <del> return nil, err } swapData = bytes.TrimSpace(swapData) // extra trailing n swapLines := strings.Split(string(swapData), "n") </del> <ins> if os.IsNotExist(err) { klog.Warningf("file %v does not exist, assuming that swap is disabled", swapFile) } else { return nil, err } } else { swapData = bytes.TrimSpace(swapData) // extra trailing n swapLines := strings.Split(string(swapData), "n") </ins> <del> // If there is more than one line (table headers) in /proc/swaps, swap is enabled and we should // error out unless --fail-swap-on is set to false. if len(swapLines) > 1 { return nil, fmt.Errorf("running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: %v", swapLines) </del> <ins> // If there is more than one line (table headers) in /proc/swaps, swap is enabled and we should // error out unless --fail-swap-on is set to false. if len(swapLines) > 1 { return nil, fmt.Errorf("running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: %v", swapLines) } </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubelet doesn't start during , journalctl shows: What you expected to happen: Kubelet correctly identifies absence of as "no swap present", kubelet service starts and can finish. How to reproduce it (as minimally and precisely as possible): Build a Linux kernel with , install kubeadm, kubelet and try . Anything else we need to know?: No registered swap devices or a kernel without swap enabled both means that there is no swap space present. Kubectl successfully checks for the former case but throws an error on the later. Failure to open means no swap. Environment: Kubernetes version (use ): Hardware configuration: OS: Kernel: Install tools:
/sig cluster-lifecycle
this is a potential improvement for the kubelet. /sig node /remove-sig cluster-lifecycle
/assign
The issue can be worked around by adding to (defined in on Debian): /etc/default/kubelet


Question: bFoundDisk := false for i, disk := range disks { <del> if disk.Lun != nil && (disk.Name != nil && diskName != "" && *disk.Name == diskName) || (disk.Vhd != nil && disk.Vhd.URI != nil && diskURI != "" && *disk.Vhd.URI == diskURI) || (disk.ManagedDisk != nil && diskURI != "" && *disk.ManagedDisk.ID == diskURI) { </del> <ins> if disk.Lun != nil && (disk.Name != nil && diskName != "" && strings.EqualFold(*disk.Name, diskName)) || (disk.Vhd != nil && disk.Vhd.URI != nil && diskURI != "" && strings.EqualFold(*disk.Vhd.URI, diskURI)) || (disk.ManagedDisk != nil && diskURI != "" && strings.EqualFold(*disk.ManagedDisk.ID, diskURI)) { </ins> // found the disk klog.V(2).Infof("azureDisk - detach disk: name %q uri %q", diskName, diskURI) disks = append(disks[:i], disks[i+1:]...)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Pods with PVs could not be moved from one node to another. According to the controller-manager logs, K8s tried to detach to Azure disks from the same node. One of the detach operation waited for this lock (). While it waited for the lock, the controller-manager pod was restarted (for whatever reason, not related to this bug, lets assume an operator restarted the pod). After the restart, K8s only tried to attach the disk to the new node but is not trying to detach it first. What you expected to happen: After a restart of the controller-manager, it should continue unfinished detach operations How to reproduce it (as minimally and precisely as possible): 2 Pods with PVs running on one node Move both pods to another node restart the controller-manager shortly after the move Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.1 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
/area provider/azure that's expected behavior, k8s controller won't continue detach volume if controller manager restarted, user needs to detach that disk volume manually.
Could you please elaborate why this behaviour is expected. I would assume that the controller manager tries to reach the desired state after a restart? Then it would need to detect that the volume is still attached to the wrong node and should try to detach it once more, before it continuing to attach the (still attached) volume? This leads to many ... Why is it required to resolve this situation by manual interaction?
I can't imagine that "manual intervention" is "desired behaviour" (yes, you wrote "expected", but that doesn't matter, if you make no proposal, how to improve this). If this isn't fixed, we can't use Azure, period. We can't have clusters that need to be babysitted / you cannot expect people running workloads on Azure clusters and babysit them and manually fixing what should/can fix itself.

Level-triggered behaviour and resilience is what made Kubernetes so successful... please bring that also to Azure's KCM/CCM.
do you know why controller manager is restarted, it's not a common scenario, we cannot guarantee disk attach/detach functionality works if controller manager restarted at that time.
related to
I think there can be many reasons why the controller-manager is restarted during running detach or attach operations e. g. the hosting machine turn into error state, if you run the controller-manager in k8s -vertical pod autoscaling,... etc. I understand you can't guarantee that attach/detach works when the controller-manager is restarted, but I don't understand why the controller-manager can't recover the situation autonomously after a restart. After restart it tries to attach the volume of the rescheduled pod and this does not work because the volume is still attached to another machine. This is indicated by a error from the arm and the controller-manager could react to it properly, right?
This is an upstream issue, and there is a discussion in , let's check whether we can fix it in cloud provider driver in that thread. Thanks.
I will try to fix this issue in azure disk driver, thanks all for the discussion. details: /assign
Thank you very much It means a lot to us, that there are no manual operations. I wonder how you guys deal with it in AKS, since Azure uses the same controllers, right? Anyway, it helps tremendously, that the clusters are self-healing.
AKS use same kube-controller-manager, while it won't be restarted easily.
OK, but how? I understand that AKS only supports 100 nodes. So, I guess you have some upper limit and just hope, nobody stresses it, because if this happens, you it either restarts or you need to scale it up and then it gets restarted as well. Also, regular maintenance, low-level as well as Kubernetes updates, all must impact it. Considering the probably thousands over thousands of clusters (I don't know, but I expect you run many), any such problem, even if unlikely by itself, must be in your interest to fix, simply because the number of clusters increases the likelihood that it happens, I would think. Anyway, thank you for making this change
Yes, that's reasonable, I will try to fix it as my top priority.


Question: ], tags = ["automanaged"], ) <ins> go_test( name = "go_default_test", srcs = ["humanreadable_test.go"], importpath = "k8s.io/kubernetes/pkg/printers", library = ":go_default_library", deps = [ "//pkg/api:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1alpha1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/runtime:go_default_library", ], ) </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: 1when error has occurwe should not print header 2 when i get ns use --all-namespaces, i get NAMESPACE column, we should return error: storageclass is not namespaced What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign
/sig cli


Question: // for dynamic provisioning exists, the driver is expected to provide // capacity information for it. CapCapacity Capability = "capacity" <ins> // To support ReadWriteOncePod, the following CSI sidecars must be // updated to these versions or greater: // - csi-provisioner:v3.0.0+ // - csi-attacher:v3.3.0+ // - csi-resizer:v1.3.0+ CapReadWriteOncePod Capability = "readWriteOncePod" </ins> ) // DriverInfo represents static information about a TestDriver.

For the ReadWriteOncePod feature to graduate to beta, we need to have E2E test coverage for its current alpha behavior. See this section of KEP-2485 for more context on what cases we need to cover for alpha: This is a requirement for beta graduation, see:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage


Question: }) } <ins> // RegisterMetrics registers a list of metrics. // This function is exported because it is intended to be used by out-of-tree plugins to register their custom metrics. func RegisterMetrics(extraMetrics ...metrics.Registerable) { for _, metric := range extraMetrics { legacyregistry.MustRegister(metric) } } </ins> // GetGather returns the gatherer. It used by test case outside current package. func GetGather() metrics.Gatherer { return legacyregistry.DefaultGatherer

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Support adding extra metrics for framework plugins Why is this needed: /sig scheduling
/assign


Question: return err } <del> // With the goal of different controllers being able to manage different // subsets of EndpointSlices, LabelManagedBy has been added to indicate // which controller or entity manages an EndpointSlice. As part of this // v1.16->v1.17 change, EndpointSlices will initially be assumed to be // managed by this controller unless a label is set to indicate otherwise. // To ensure a seamless upgrade process, the managedBySetupAnnotation is // used to indicate that LabelManagedBy has been set initially for related // EndpointSlices. If it hasn't been set to the expected value here, we call // ensureSetupManagedByAnnotation() to set up LabelManagedBy on each // EndpointSlice. // TODO(robscott): Remove this before v1.18. err = c.ensureSetupManagedByAnnotation(service) if err != nil { c.eventRecorder.Eventf(service, v1.EventTypeWarning, "FailedToSetEndpointSliceManagedByLabel", "Error adding managed-by Label to Endpoint Slices for Service %s/%s: %v", service.Namespace, service.Name, err) return err } </del> esLabelSelector := labels.Set(map[string]string{ discovery.LabelServiceName: service.Name, discovery.LabelManagedBy:   controllerName,

What happened: kube-controller-manager panic for What you expected to happen: no panic. How to reproduce it (as minimally and precisely as possible): No idea. I guess it's hard to reproduce. Anything else we need to know?: Found from here: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign cc Should we do something for v1.17? (v1.17 code freeze now, but still have the opportunity to fix) can be removed safely in v1.18, right?
/sig scheduling
Working on this in /sig network /priority critical-urgent /assign
Sorry, I didn't realize it. I should have a search before firing an issue. I recommend adding this issue to your PR. e.g. `
Updated, thanks!
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This seems to be writing to a cached (should be const) service. I think we need to DeepCopy() it before we mutate the Go type. does that sound right? Is this still our best state of the art?
yes, copy before change
we have a tool to catch this (the cache mutation detector), but it got disabled in our test runs two years ago (xref )
yes great catch - I was always convinced this is enabled...


Question: informerFactory.Start(stop) go hpaController.Run(stop) <ins> // Wait for HPA to be processed. <-tc.processed </ins> tc.Lock() <ins> tc.finished = true </ins> if tc.verifyEvents { tc.Unlock() // We need to wait for events to be broadcasted (sleep for longer than record.sleepDuration).

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Resync period for HorizontalPodAutoscaler controller is set to 15 seconds in order to update HPAs every 15 seconds. Sometimes updates are skipped and period between two updates is 30 seconds. What you expected to happen: HPA should be updated every 15 seconds. How to reproduce it (as minimally and precisely as possible): Add debugs to HPA controller showing how often HPAs are processed. Create HPA. Anything else we need to know?: HPA controller uses rate RateLimitingQueue: New requests are always with 15 seconds delay: If request for the given HPA is in the queue then newly request is skipped. Requests are and removed from the queue at the same moment so there is a race condition. See following scenario: 00:00:00 - Resync adds request to queue to be executed at 00:00:15 00:00:15 - Resync adds next request to queue to be executed at 00:00:30 but it is skipped because previous request is still in the queue 00:00:15 - Request at 00:00:00 is executed 00:00:30 - Resync adds next request to queue to be executed at 00:00:45 00:00:45 - Request at 00:00:30 is executed In above scenario HPA wasn't updated between 00:00:15 and 00:00:45. Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration: GKE <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig autoscaling
/assign krzysztof-jastrzebski


Question: } var nodeCIDRMaskSizeIPv4, nodeCIDRMaskSizeIPv6 int <del> if utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) { </del> <ins> if dualStack { </ins> // only --node-cidr-mask-size-ipv4 and --node-cidr-mask-size-ipv6 supported with dual stack clusters. // --node-cidr-mask-size flag is incompatible with dual stack clusters. nodeCIDRMaskSizeIPv4, nodeCIDRMaskSizeIPv6, err = setNodeCIDRMaskSizesDualStack(ctx.ComponentConfig.NodeIPAMController) } else { // only --node-cidr-mask-size supported with single stack clusters. <del> // --node-cidr-mask-size-ipv4 and --node-cidr-mask-size-ipv6 flags are incompatible with dual stack clusters. </del> <ins> // --node-cidr-mask-size-ipv4 and --node-cidr-mask-size-ipv6 flags are incompatible with single stack clusters. </ins> nodeCIDRMaskSizeIPv4, nodeCIDRMaskSizeIPv6, err = setNodeCIDRMaskSizes(ctx.ComponentConfig.NodeIPAMController) }

With the feature gate disabled, kube-controller-manager accepts a argument, but with enabled, it only accepts and and rejects . This blocks moving dual-stack to beta, since switching the feature gate to on-by-default would break existing single-stack clusters using . /cc /sig network /priority important-soon
let me tag too, IIRC he implemented it and may be is more familiar with this
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved


Question: "k8s.io/apiserver/pkg/authentication/request/bearertoken" bootstrapapi "k8s.io/cluster-bootstrap/token/api" "k8s.io/kubernetes/plugin/pkg/auth/authenticator/token/bootstrap" <del> bootstraputil "k8s.io/kubernetes/test/e2e/lifecycle/bootstrap" </del> "k8s.io/kubernetes/test/integration" "k8s.io/kubernetes/test/integration/framework" )

Originally tracked at . After fixing "not include test files in import-boss" in , we have a regression here: is using . We need remove dependencies from integration tests to e2e tests again. /assign /sig testing
/sig auth as per the required change
/kind cleanup /area test
It would be helpful to outline an argument as to why this is the case. I can imagine reasons, but none are mentioned here.
Description updated.
/triage accepted /priority backlog /milestone v1.21 This seems a simple enough fix to land (and thank you for doing this) but marking low enough priority to make it clear we may back out and wait for v1.22 if we don't have bandwidth for this
So while I think we should probably do this anyhow I want to note: This issue lists the reasons: we have no intention to stage e2e (there's been much discussion about this since then) we generally don't want anything non-informational (required jobs) running only sometimes anyhow, it's just not reliable Decoupling these seems like a good idea from a purely code-organization perspective, but the reasons in the previous issue are no longer valid.
/milestone v1.22 The PR didn't land in time. At this point I view this as nice to have. Even if it's low impact, let's wait until next release.


Question: continue } <del> if previousOp.podName != emptyUniquePodName && podName != emptyUniquePodName && </del> <ins> if previousOp.podName != EmptyUniquePodName && podName != EmptyUniquePodName && </ins> previousOp.podName != podName { // No match, keep searching continue

During testing a AWS cluster, found AWS volume once was pending for detach about 30 minutes. According to the log, the following happened, controller issues a detach operation after detach response returns, it periodically checks the status until it reaches "detached". However, in this test, the status is "busy" all the time. after reaching the timeout 30 mins, detach function returns error because the status is not "detached" The following is the log related to the failed test I0915 18:29:46. 5 ] Verified volume is safe to detach for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-". E0915 18:59:54. 5 ] Error detaching volumeID "vol-": Timeout waiting for volume state: actual=busy, desired=detached E0915 18:59:54. 5 ] Operation for """" failed. No retries permitted until 2016-09-15 18:59:55. +0000 UTC (durationBeforeRetry 500ms). Error: DetachVolume.Detach failed for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-" with: Timeout waiting for volume state: actual=busy, desired=detached
Thoughts on what the ideal timeout should be. Too short would be bad because it would give a false-negative.
I know this is from a test but I'm seeing the same problem on a production cluster (1.4.8). I'm not sure exactly what triggers it yet. Kubelet only unmounts the pod directory, not the mount in "aws-ebs/mounts". As a result, the EBS gets stuck in "busy" state when detaching since it is still mounted to the VM. Manually unmounting the EBS allows it to get unstuck and detach safely. In controller manager I see: It then proceeds to get stuck detaching (volume goes to busy state), after which it starts trying to force detach which of course hangs indefinitely.

Inspecting the node directly, I see it's still mounted which is why detach hangs: So something in kubelet is messed up and not unmounting both mount points correctly.
I think this might be the same issue as
On kubelet, it looks like the volume failed to mark as mounted, which is why it subsequently got stuck. This may have to do with the fact this pod () got rolled back while waiting for the previous deployment to succeed.
^ Looks like it might be a race condition between the controller manager and reconciler: manager attaches volume starts attachment process (pod rollback/delete) controller manager thinks its safe to detach, so removes volume from node's volume list, then tries to detach mounts the volume as part of the ongoing attach/mount process from (2) tries to update the mount list, this fails because something (reconciler?) has removed the volume from attached list gets stuck in (busy) state detaching, as its mounted to the kubelet node. it never gets unmounted again in kubelet because state doesn't reflect it is mounted.
Looks like there exists race conditions between external updates of and 's asynchronous kick off of operations like MountVolume, which is the case I'm seeing. Seems like a lot of these edge cases could crop up here - maybe reconciler should lock changes to the desiredStateOfWorld while it's running?
thank you for providing detailed information and analysis. I think there is a race condition between asynchronous operation MountVolume and updating actualStateOfWorld directly by reconciler I will work on a fix. Please let me know if you have any issue/comments.


Question: <del> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package mount import ( "fmt" "os" "syscall" "k8s.io/klog" ) // CleanupMountPoint unmounts the given path and // deletes the remaining directory if successful. // if extensiveMountPointCheck is true // IsNotMountPoint will be called instead of IsLikelyNotMountPoint. // IsNotMountPoint is more expensive but properly handles bind mounts within the same fs. func CleanupMountPoint(mountPath string, mounter Interface, extensiveMountPointCheck bool) error { // mounter.ExistsPath cannot be used because for containerized kubelet, we need to check // the path in the kubelet container, not on the host. pathExists, pathErr := PathExists(mountPath) if !pathExists { klog.Warningf("Warning: Unmount skipped because path does not exist: %v", mountPath) return nil } corruptedMnt := IsCorruptedMnt(pathErr) if pathErr != nil && !corruptedMnt { return fmt.Errorf("Error checking path: %v", pathErr) } return doCleanupMountPoint(mountPath, mounter, extensiveMountPointCheck, corruptedMnt) } // doCleanupMountPoint unmounts the given path and // deletes the remaining directory if successful. // if extensiveMountPointCheck is true // IsNotMountPoint will be called instead of IsLikelyNotMountPoint. // IsNotMountPoint is more expensive but properly handles bind mounts within the same fs. // if corruptedMnt is true, it means that the mountPath is a corrupted mountpoint, and the mount point check // will be skipped func doCleanupMountPoint(mountPath string, mounter Interface, extensiveMountPointCheck bool, corruptedMnt bool) error { if !corruptedMnt { var notMnt bool var err error if extensiveMountPointCheck { notMnt, err = mounter.IsNotMountPoint(mountPath) } else { notMnt, err = mounter.IsLikelyNotMountPoint(mountPath) } if err != nil { return err } if notMnt { klog.Warningf("Warning: %q is not a mountpoint, deleting", mountPath) return os.Remove(mountPath) } } // Unmount the mount path klog.V(4).Infof("%q is a mountpoint, unmounting", mountPath) if err := mounter.Unmount(mountPath); err != nil { return err } notMnt, mntErr := mounter.IsLikelyNotMountPoint(mountPath) if mntErr != nil { return mntErr } if notMnt { klog.V(4).Infof("%q is unmounted, deleting the directory", mountPath) return os.Remove(mountPath) } return fmt.Errorf("Failed to unmount path %v", mountPath) } // TODO: clean this up to use pkg/util/file/FileExists // PathExists returns true if the specified path exists. func PathExists(path string) (bool, error) { _, err := os.Stat(path) if err == nil { return true, nil } else if os.IsNotExist(err) { return false, nil } else if IsCorruptedMnt(err) { return true, err } else { return false, err } } // IsCorruptedMnt return true if err is about corrupted mount point func IsCorruptedMnt(err error) bool { if err == nil { return false } var underlyingError error switch pe := err.(type) { case nil: return false case *os.PathError: underlyingError = pe.Err case *os.LinkError: underlyingError = pe.Err case *os.SyscallError: underlyingError = pe.Err } return underlyingError == syscall.ENOTCONN || underlyingError == syscall.ESTALE || underlyingError == syscall.EIO || underlyingError == syscall.EACCES } </del>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: pod with smb(cifs) mount could not be terminated on Windows when that smb account is invalid, the pod will be stuck forever What you expected to happen: How to reproduce it (as minimally and precisely as possible): a pod with azure file mount on Windows the storage account of azure file invalid the pod with azure file mount could not be terminated forever Anything else we need to know?: similar to case: Environment: Kubernetes version (use ): v1.12.6 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /assign /sig windows


Question: <ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e import ( "fmt" "strconv" "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) var _ = Describe("Events", func() { var c *client.Client BeforeEach(func() { c = loadClientOrDie() }) It("should be sent by kubelets and the scheduler about pods scheduling and running", func() { provider := testContext.provider if len(provider) > 0 && provider != "gce" && provider != "gke" && provider != "aws" { By(fmt.Sprintf("skipping TestKubeletSendsEvent on cloud provider %s", provider)) return } podClient := c.Pods(api.NamespaceDefault) By("creating the pod") name := "send-events-" + string(util.NewUUID()) value := strconv.Itoa(time.Now().Nanosecond()) pod := &api.Pod{ ObjectMeta: api.ObjectMeta{ Name: name, Labels: map[string]string{ "name": "foo", "time": value, }, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  "p", Image: "kubernetes/serve_hostname", Ports: []api.Port{{ContainerPort: 80, HostPort: 8080}}, }, }, }, } By("submitting the pod to kubernetes") _, err := podClient.Create(pod) if err != nil { Fail(fmt.Sprintf("Failed to create pod: %v", err)) } defer func() { By("deleting the pod") defer GinkgoRecover() podClient.Delete(pod.Name) }() By("waiting for the pod to start running") waitForPodRunning(c, pod.Name) By("verifying the pod is in kubernetes") pods, err := podClient.List(labels.SelectorFromSet(labels.Set(map[string]string{"time": value}))) Expect(len(pods.Items)).To(Equal(1)) By("retrieving the pod") podWithUid, err := podClient.Get(pod.Name) if err != nil { Fail(fmt.Sprintf("Failed to get pod: %v", err)) } fmt.Printf("%+vn", podWithUid) // Check for scheduler event about the pod. By("checking for scheduler event about the pod") events, err := c.Events(api.NamespaceDefault).List( labels.Everything(), labels.Set{ "involvedObject.kind":      "Pod", "involvedObject.uid":       string(podWithUid.UID), "involvedObject.namespace": api.NamespaceDefault, "source":                   "scheduler", }.AsSelector(), ) if err != nil { Fail(fmt.Sprintf("Error while listing events:", err)) } Expect(len(events.Items)).ToNot(BeZero(), "scheduler events from running pod") fmt.Println("Saw scheduler event for our pod.") // Check for kubelet event about the pod. By("checking for kubelet event about the pod") events, err = c.Events(api.NamespaceDefault).List( labels.Everything(), labels.Set{ "involvedObject.uid":       string(podWithUid.UID), "involvedObject.kind":      "BoundPod", "involvedObject.namespace": api.NamespaceDefault, "source":                   "kubelet", }.AsSelector(), ) if err != nil { Fail(fmt.Sprintf("Error while listing events:", err)) } Expect(len(events.Items)).ToNot(BeZero(), "kubelet events from running pod") fmt.Println("Saw kubelet event for our pod.") }) }) </ins>

Create the ginkgo based events test suite and put the following test in it: TestKubeletSendsEvent
Part of the ginko-ization of the e2e tests


Question: <ins> #!/bin/bash set -e set -x CGO_ENABLED=0 go build  -a -ldflags '-extldflags "-static" -s' hello.go </ins>

/kind feature Hey, in K8s 1.23 we are planning to deprecate multiple klog flags. Deprecated flags: , , , , , , , , , , , One of the alpha milestones is requires us to remove usage of deprecated flags within K8s repo. Examples found for kubemark [x] [x] [x] [x] Recommendation, either: Stop depending on writing to files and remove flags , or Add go-runner to kubemark image and migrate to /sig scalability /wg structured-logging
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Resolved in both k/k and kubernetes-sigs/cloud-provider-azure. /close


Question: // Represents time when the job was completed. It is not guaranteed to // be set in happens-before order across separate operations. // It is represented in RFC3339 form and is in UTC. <ins> // The completion time is only set when the job finishes successfully. </ins> // +optional optional k8s.io.apimachinery.pkg.apis.meta.v1.Time completionTime = 3;

The API docs for jobs are unclear as to what fields a caller would use to determine job success or failure in 1.11 (and we haven't updated them in a while) A casual reader would infer that completionTime is set when the job fails (but it isn't). Linking out to the docs is ok, but I shouldn't have to click through the links to understand how to use the Job object. The Job description and the job status fields should clearly indicate what fields are set in completion states on the job, because that's 100% of the point of docs. Taking a summary of the jobs-run-to-completion link and putting it in , , and job would improve the actual utility of our API doc. We should also do a pass on other objects and make the actual in server docs more useful.
as a consumer of the API, I had to read way too much to know how to use jobs correctly and how to write an API client that properly handled the conditions. Also, we should consider either setting completionTime on failures, or adding failedTime (the former may not be forwards compatible), because reading the conditions is too complex for a fundamental use case like this. This matches our decision w.r.t. conditions vs. fields - conditions are useful for extension and extra data, but status should have fields that allow an API consumer to easily understand state.
Also... our docs don't actually explain how to determine the job is complete via the API, so there's arguably no place this is captured.
I have the same problem right now, of correctly identifying a finished job, when it does not complete successfully and after reading the documentation and the reference and all the examples I could find, it's still not clear how I can check the conditions and make sure the job is finished when I do so: -what does each mean? can I rely on , is that set only when the job finishes, whether via or ? what other reasons are possible? It would definitely help to have specified on failure as well, and the differentiation to be made via the presence of .
/assign soltysh
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I have to say this issue is still very relevant. As of now, it is very hard to design something around Job Statuses. Anyway to move this forward?
Completely agree with previous comments! Even I found the way how to detect completed/failed jobs, still not sure am I covered all use cases. This part definitely needs clarifications/improvements/better documentation.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
With regards to: If you actually follow this link and then search on that page for "conditions" you will get exactly zero hits. This is extremely unhelpful.
This documentation really, really needs work. (So does the API, for that matter.)
So what is the correct way to determine if a job is finished and whether it completed successfully?
When a Job completes (finished successfully), its will be set, and Job conditions will have one with . When a Job fails, Job conditions will have one with .
Hi, if nobody has been working on it, can I take this one? I have gone through the issue which is mentioned here and I would like to work on this. Can you please help me to get started as I am beginner to the project.
I totally agree with this- currently the only way to get the full picture of a Job is to read its conditions along with status fields which is error prone and a lot of extra work. I think a good start would be to add a new field as suggested by so as to not break compatibility for users already depending on
This sound reasonable. I'm still unclear when can contain more than one element. Also what does a or mean?
Still, open -- anyone actively working on this? I would like to attempt if not. :)
Are you still working on this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
bump
This does not match what I just witnessed in my cluster… digging deeper to see if I need to file a bug but I have a Job with a single Pod that was executed. The Pod failed but the Job was marked as complete. This is the one (and only) pod that executed as my job: And this is the job that owns it (from kubectl describe): This is the job status object: on v1.16.7 but not finding any similar reports. just found this page as my only relevant search result.
I think it not only needs documentation, but also a change: It would be good to have 2 fields (probably needs better names), this way it's easy and clear when a job is finished and if it failed or not and when (someone can look at failed, succeeded and conditions for details): finishedtime: only set when the job finished (either completely failed or completely finished) finishtype: 'failed or completed'


Question: // Obtain Load Balancer Ingress IPs var ips []string for _, ing := range service.Status.LoadBalancer.Ingress { <del> ips = append(ips, ing.IP) </del> <ins> if ing.IP != "" { ips = append(ips, ing.IP) } </ins> } if len(ips) > 0 {

Originally post on project <!-- Thanks for helping us to improve K3S! We welcome all bug reports. Please fill out each area of the template so we can better help you. Comments like this will be hidden when you post but you can delete them if you wish. --Environmental Info: K3s Version: v1.21.0+k3s1 <!-- Provide the output from "k3s -v" --Node(s) CPU architecture, OS, and Version: Linux 4.19.0-16-cloud-amd64 SMP Debian 4.19.181-1 (2021-03-19) x86_64 <!-- Provide the output from "uname -a" on the node(s) --Cluster Configuration: 1 server & 3 agents <!-- Provide some basic information on the cluster configuration. For example, "3 servers, 2 agents". --Describe the bug: I use an external cloud controller manager to manage load balancing and services exposition. But on my services spec, I have no field but a instead. Everything works fine but server logs are flooded with entries like <!-- A clear and concise description of what the bug is. --Steps To Reproduce: <!-- Steps to reproduce the behavior. Please include as the first step how you installed K3s on the node(s) (including all flags or environment variables). If you have customized configuration via systemd drop-ins or overrides () please include those as well. --- Installed K3s: Use an external CCM which only provide a hostname to expose service Expected behavior: No more logs like <!-- A clear and concise description of what you expected to happen. --Actual behavior: Server logs are flooded with entries like <!-- A clear and concise description of what actually happened. --Additional context / logs: <!-- Add any other context and/or logs about the problem here. --Should it be a good idea to test consistency of IP here : or here ?
/sig cloud-provider
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg structured-logging Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/assign
could you help to triage this issue? Since still have label
/triage accepted Seems like a missing check for ""


Question: if pvc.Spec.VolumeMode != nil { w.Write(LEVEL_0, "VolumeMode:t%vn", *pvc.Spec.VolumeMode) } <ins> if pvc.Spec.DataSource != nil { w.Write(LEVEL_0, "DataSource:n") w.Write(LEVEL_1, "Name:t%vn", pvc.Spec.DataSource.Name) w.Write(LEVEL_1, "Kind:t%vn", pvc.Spec.DataSource.Kind) } </ins> printPodsMultiline(w, "Mounted By", mountPods) if len(pvc.Status.Conditions) > 0 {

Currently there's now way for a user to check what they entered in the DataSource field of a PVC create request. In the past, this wasn't as confusing because if one were to submit invalid info (unsupported Kind) the verification checks would fail and the request would be rejected with a helpful message indicating why. As of 1.14 that's no longer the case, a user is able to put anything they wish in these fields and the create is performed and they get a PVC, and there's no way for them to see what is or isn't set. Adding output of DataSource to would be useful, not only for invalid cases where the input might just get eaten, but also for users to have the ability to check for originating sources of their PVCs.
/sig storage
/assign
GitHub didn't allow me to assign the following users: me. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten


Question: kubectl set resources --dry-run --local -f ${fluentd_gcp_yaml}  --limits=memory=${FLUENTD_GCP_MEMORY_LIMIT}  --requests=cpu=${FLUENTD_GCP_CPU_REQUEST},memory=${FLUENTD_GCP_MEMORY_REQUEST}  <del> -o yaml > ${fluentd_gcp_yaml}.tmp </del> <ins> --containers=fluentd-gcp -o yaml > ${fluentd_gcp_yaml}.tmp </ins> mv ${fluentd_gcp_yaml}.tmp ${fluentd_gcp_yaml} }

I'm consistently seeing the following failure in those runs: Seems like they're failing scheduling. Will look into it. cc
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Ok.. So it's consistently failing to create 6 pods and almost surely it's an issue with requests. Most probably some node-daemon is requesting more than before.
From the I'm almost 100% sure that it's failing due to this PR - It seems to have increased the fluentd memory request. Could one of you confirm this?
It hasn't (shouldn't have) changed the resource requirement, defaults for the env variable are the same as in the yaml
Cool, thanks. I'll check what else it could've been.
So it's due to insufficient cpu:
Unless kubemark was using another value than set in that yaml? At some point in time the cpu request was lower ().
It was almost immediately reverted:
We need 50m (kubelet) + 20m (proxy) + 20m (npd) for each hollow-node. And all nodes seem very close to being exhausted: Do you by any chance remember how much was used before?
The above pr should fix this
So the problem is indeed with fluentd. It's requesting 200m of CPU instead of 100m (as is expected). The fluentd container requests 100m as it should be - But the prometheus-to-sd-exporter sidecar container has also requested 100m : We don't have any requests for the exporter set in the yaml template - But it's being set.
There is a limitranger that sets default request to 100m but that's in default namespace which shouldn't affect fluentd. So it's indeed a bug with your PR Could you fix it?
Now these numbers seems awfully similar to the fluentd numbers. Now I see the link with Please investigate, it's a huge bug and has to be rolled back!
Testing a fix, I will send a PR soon.
Seems fixed - https://k8s-


Question: return cfg } <del> func newFackeAzureToken(accessToken string, expiresOn string) adal.Token { </del> <ins> func newFakeAzureToken(accessToken string, expiresOnTime time.Time) *azureToken { return &azureToken{ token:       newFakeADALToken(accessToken, strconv.FormatInt(expiresOnTime.Unix(), 10)), environment: "testenv", clientID:    "fake", tenantID:    "fake", apiserverID: "fake", } } func newFakeADALToken(accessToken string, expiresOn string) adal.Token { </ins> return adal.Token{ AccessToken:  accessToken, RefreshToken: "fake",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, when AAD login is configured, refresh token timeout would make kubectl print an error, not able to re-login without manually remove the tokens. Expect kubectl to promote user to login again. Why is this needed: Configure refresh-token timeout from Azure is a way to force re-login after inactivity. kubectl doesn't promote re-login makes users harder to apply timeout policy.
/sig azure
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig platform/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure
the root cause of this issue is in this first gets the cached token (result of device code flow) at it's trying to refresh an expired token at and failed. An error is returned. Since the expired token is still in the cache, it will always hit this path until user removes the token from kubeconfig. the fix should be: the token from cache a retry to re-initiate the device code flow in
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig platform/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
I'm working on it.
are you working on it? I will grab it if I dont hear it from you :)
have the code ready will try PR this weekend Best, Ping
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Closing as merged


Question: set -e SSH_OPTS="-oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oLogLevel=ERROR -C" <ins> source "${KUBE_ROOT}/cluster/common.sh" </ins> MASTER="" MASTER_IP=""

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: I was following to install a cluster but failed due to the following. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
The current workaround solution is remove in ./kube-
please have a look
thanks for the quick response
Just FYI, I think that you should fix both kube-up and kube-down
i have already move: source "${KUBE_ROOT}" to the head of ,is there something can't work?
Thanks, I did not have a try but just a reminder, just check code, yes, I think it works. Thanks
Seems the kube-up still does not work, filed another issue here
can you help take a look? Thanks!
/sig cluster-lifecycle
hi, I still see the issue but wondering since you have closed considering planned deprecation of the script. Does this issue still requires a fix or any decision was made not to work on it? Thanks!
Thanks , I think that we can close this now as it is not supported any more.
sure and thank you for the quick response and closing it !!


Question: "        ExecStart=/opt/bin/kube-apiserver n", "        --address=0.0.0.0 n", "        --port=8080 n", <del> "        --portal_net 10.244.0.0/16 n", </del> <ins> "        --portal_net=10.244.0.0/16 n", </ins> "        --etcd_servers=http://127.0.0.1:4001 n", "        --public_address_override=$private_ipv4 n", "        --logtostderr=truen",

The issue is a missing = in the arguments for starting kube-apiserver. Fix at: in branch:


Question: GetStats(path string, stats *libcontainercgroups.Stats) error } <del> // getSupportedSubsystems returns list of subsystems supported func getSupportedSubsystems() []subsystem { supportedSubsystems := []subsystem{ &cgroupfs.MemoryGroup{}, &cgroupfs.CpuGroup{}, </del> <ins> // getSupportedSubsystems returns a map of subsystem and if it must be mounted for the kubelet to function. func getSupportedSubsystems() map[subsystem]bool { supportedSubsystems := map[subsystem]bool{ &cgroupfs.MemoryGroup{}: true, &cgroupfs.CpuGroup{}:    true, </ins> } <ins> // not all hosts support hugetlb cgroup, and in the absent of hugetlb, we will fail silently by reporting no capacity. </ins> if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.HugePages) { <del> supportedSubsystems = append(supportedSubsystems, &cgroupfs.HugetlbGroup{}) </del> <ins> supportedSubsystems[&cgroupfs.HugetlbGroup{}] = false </ins> } if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.SupportPodPidsLimit) { <del> supportedSubsystems = append(supportedSubsystems, &cgroupfs.PidsGroup{}) </del> <ins> supportedSubsystems[&cgroupfs.PidsGroup{}] = true </ins> } return supportedSubsystems }

If I run first time it goes fine (after reboot). For subsequent runs I get nodes NotReady. I've tried restarting docker but that won't help. Before christams this used to work all the time. Running Arch Linux if it helps. In I see
/sig node
ditto for me: Looking for a work around I noticed that HugePages recently moved to beta. I can get pods to start again by disabling the HugePages feature gate .
/assign
cc
default Arch kernel doesn't have a hugetlb cgroup controller compiled in(been there done that, I use Arch as well), so you have to prepare your own kernel to run it.
If it were a kernel problem I would have expected the kubelet to fail the first time it was started rather than the second time. This isn't confined to Arch, though. I'm having the same problem with the default Debian kernel:
can you check if your Kernel was compiled with flag? You can do it by running: I bet we have to add documentation, describing this behavior. On Ubuntu 16.04 machine:
Sure, looks like it was not: Documentation (especially in the error message) would be great, but if the kubelet isn't going to survive a restart, I would prefer that it not start the first time. Better to fail immediately than eventually.
can you please check if works (as in fails!) for you?
No, this doesn't resolve the issue. Logs from the first start: Logs from the second start:
Do you have the directory? in that case what i have is insufficient :( If you don't have that directory, then we should probably look for a better spot to put that check? any advice?
No, that directory is missing on my system:
I'm seeing this issue as well.
I am preparing a fix to not fail when hugetlb mount is not present. Will post a fix shortly.
I posted a fix here:


Question: {"baz", false}, // Missing local prefix. {v1.AppArmorBetaProfileNamePrefix + "/usr/sbin/ntpd", true}, {v1.AppArmorBetaProfileNamePrefix + "foo-bar", true}, <del> {v1.AppArmorBetaProfileNamePrefix + "unloaded", false}, // Not loaded. {v1.AppArmorBetaProfileNamePrefix + "", false}, </del> } for _, test := range tests { <del> err := validateProfile(test.profile, loadedProfiles) </del> <ins> err := ValidateProfileFormat(test.profile) </ins> if test.expectValid { assert.NoError(t, err, "Profile %s should be valid", test.profile) } else {

What happened: Pods with invalid container-level seccomp profiles allow its init containers to be executed. Pods with unloaded apparmor profiles blocks the entire pod. What you expected to happen: Consistency between both. In one hand AppArmor blocks all pod execution regardless of the containers having or not a profile attached to it. On the other, Seccomp allows for an init container to deploy its own profile - as currently done on . How to reproduce it (as minimally and precisely as possible): Apply the yaml below: Resulting in only the init container from the being executed: ! Anything else we need to know?: AppArmor is currently being target to GA on 1.21 (). Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.0-alpha.0.292+", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-12-12T23:13:21Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node /area security
/cc
/sig security /triage accepted
I think we can consider this as a bug, right? Then we could work on a fix for v1.21 interdependently of the AppArmor GA graduation.
Sounds like a good plan :+1:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/assign


Question: t.lock.Lock() defer t.lock.Unlock() <del> found := false for i, existingObj := range t.objects[gvr] { objMeta, err := meta.Accessor(existingObj) if err != nil { return err } if objMeta.GetNamespace() == ns && objMeta.GetName() == name { obj := t.objects[gvr][i] t.objects[gvr] = append(t.objects[gvr][:i], t.objects[gvr][i+1:]...) for _, w := range t.getWatches(gvr, ns) { w.Delete(obj) } found = true break } </del> <ins> objs, ok := t.objects[gvr] if !ok { return errors.NewNotFound(gvr.GroupResource(), name) </ins> } <del> if found { return nil </del> <ins> namespacedName := types.NamespacedName{Namespace: ns, Name: name} obj, ok := objs[namespacedName] if !ok { return errors.NewNotFound(gvr.GroupResource(), name) </ins> } <del> return errors.NewNotFound(gvr.GroupResource(), name) </del> <ins> delete(objs, namespacedName) for _, w := range t.getWatches(gvr, ns) { w.Delete(obj) } return nil </ins> } // filterByNamespace returns all objects in the collection that // match provided namespace. Empty namespace matches // non-namespaced objects. <del> func filterByNamespace(objs []runtime.Object, ns string) ([]runtime.Object, error) { </del> <ins> func filterByNamespace(objs map[types.NamespacedName]runtime.Object, ns string) ([]runtime.Object, error) { </ins> var res []runtime.Object for _, obj := range objs {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I used to fake a clientset to test my code. I found that it became very slow to init the clientset with a big number of objects when I was trying to simulate a large scale cluster, and the time seemed exponential increasing. For example: Initializing a clientset with: 5000 objects took 300ms objects took 2.58s objects took 11.82s objects took 54.64s. It's because the fake clientset used a slice to store each kind of objects, so it's quite slow to init the clientset with massive objects because it checked existence of an object by traversing all objects and comparing with them before adding one, which leads to O(n^2) time complexity. What you expected to happen: The fake clientset is very useful for development, it should have a reasonable time complexity to allow developers to use it for large scale scenario. How to reproduce it (as minimally and precisely as possible): Run this code: You would see the time spent on each operation, like: Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.0-1.18.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I had PR that uses map to store objects, which can reduce the init time to a reasonable value (tens of ms for objects). /assign


Question: }, } } <del> // A object with x-kubernetes-preserve-unknown-fields but no properties or additionalProperties is treated // as an empty object. if schema.IsXPreserveUnknownFields() { return &unstructuredMap{ value:  m, schema: schema, propSchema: func(key string) (Schema, bool) { return nil, false }, } </del> <ins> // properties and additionalProperties are mutual exclusive, but nothing prevents the situation // where both are missing. // An object that (1) has no properties (2) has no additionalProperties or additionalProperties == false // is treated as an empty object. // An object that has additionalProperties == true is treated as an unstructured map. // An object that has x-kubernetes-preserve-unknown-field extension set is treated as an unstructured map. // Empty object vs unstructured map is differentiated by unstructuredMap implementation with the set schema. // The resulting result remains the same. return &unstructuredMap{ value:  m, schema: schema, propSchema: func(key string) (Schema, bool) { return nil, false }, </ins> } <del> return types.NewErr("invalid object type, expected either Properties or AdditionalProperties with Allows=true and non-empty Schema") </del> } if schema.Type() == "array" {

While migrating webhooks to CEL for one of the structs in our CRD api, I noticed that the empty object in CEL is interpreted wrongly. Empty map object seems to be interpreted correctly. In the example below, both map and object are sent empty () in json output. returns true json (captured on the client side) sent to the apiserver CRD schema No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
CEL only sees the JSON sent to the server, and is correctly evaluated as present. The issue is with the client sending the request populating a value in that field. How is the request being constructed? /remove-kind bug /kind support
/cc /triage accepted
Upon investigation, the problem lies there: We requires either or be set.
Thanks for raising this issue! I was debugging offline with and seems like CRD/CR validation didn't handle the situation of unsetting properties with field type specified . It is a corner case in CRD since we normally would ask for properties defined if the field type is (I could not think of use case which having properties unset). But it might ok to allow it through since it won't hurt from CRD perspective. We will fix in CEL values handling which treat this as empty map passed. /assign
GitHub didn't allow me to assign the following users: jiahui. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : Thanks for raising this issue! I was debugging offline with and seems like CRD/CR validation didn't handle the situation of unsetting properties with field type specified 'Object1object`(I could not think of use case which having properties unset). But it might ok to allow it through since it won't hurt from CRD perspective. We will fix in CEL values handling which treat this as empty map passed. /assign </details
also check if the CRD validation allows the or values, and if so, if the CEL code handles those properly as well
FYI: We do not allow in crd structured schema now:


Question: autoscalingv1 "k8s.io/api/autoscaling/v1" autoscalingv2 "k8s.io/api/autoscaling/v2beta2" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/meta/testrestmapper" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

What happened: I setup custom metrics based autoscaling with multiple metrics. One of those metrics was not available. HPA reports all metrics as "unknown" (even the ones that are still available) and refuses to operate. What you expected to happen: Pods would be automatically scaled based on metrics that were available, ignoring missing metrics (treating them as zero). A partial failure in the metrics system should not prevent auto-scaling from proceeding with the data that it has. For example, CPU % metrics come from the k8s metrics server whereas the other metrics come from the prometheus adapter. If the prometheus adapter goes away we can still use the CPU metric as a lower bound on the number of replicas. If you consider, e.g. , instead of aborting the whole metric gathering process there, it could have stored a placeholder or zero metric value when it fails to fetch the metric. How to reproduce it (as minimally and precisely as possible): Setup a custom metrics server and tell HPA to scale based on several metrics, one of which is not actually available. Environment: kops 1.8 on AWS Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.8", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:23:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
/sig autoscaling
cc
/kind bug /priority important-longterm
this return returns nil value of statuses variable. statuses variable contains the value of all metrics.
as long as we failed to get one metric of all metrics, the func computeReplicasForMetrics returns, the scale lose efficacy, it is not highly available. when failed to get metric, should we record this event and do not return:)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still having this problem.
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this issue was resolved or it works as designed? thanks,
I was testing this last week and it is not resolved. I don't think it is reasonable to say this is working as designed.
Seems there is a PR to fix it, but it hasn't been reviewed or merged.
yes, I looked into the master branch code, not fixed yet, just curious about it, it would not be fixed or as dobesv said it will be merged/fixed latter with that PR.

/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten /lifecycle frozen /priority backlog
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten /lifecycle frozen /priority backlog Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue was discussed at SIG Autoscaling yesterday: I'm going to raise a new PR later this week given the code's changed so much since the original PR was raised by that it doesn't make sense to just rebase it.
Thanks, that would be excellent!


Question: <ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kubeadm import ( "bytes" "fmt" "io" "os" "os/exec" ) // Forked from test/e2e/framework because the e2e framework is quite bloated // for our purposes here, and modified to remove undesired logging. func RunCmd(command string, args ...string) (string, string, error) { var bout, berr bytes.Buffer cmd := exec.Command(command, args...) cmd.Stdout = io.MultiWriter(os.Stdout, &bout) cmd.Stderr = io.MultiWriter(os.Stderr, &berr) err := cmd.Run() stdout, stderr := bout.String(), berr.String() if err != nil { return "", "", fmt.Errorf("error running %s %v; got error %v, stdout %q, stderr %q", command, args, err, stdout, stderr) } return stdout, stderr, nil } </ins>

go test ./cmd/kubeadm/... fails with: it appears to be following a "RunCmd" pattern used by e2e tests, but unit tests should not require built binaries
can work on this
has a long discussion as to why this is. These tests only get run when gets run, which will produce the needed binary and pass it in as arg. For more information, authored this and should be able to provide more details.
Yup, the target explicitly ignores these tests: and like said, runs this script as part of its target:
then the test should have build tags to exclude it from a regular run of the package
or up for fixing that?
I thought build tags only works on the OS level or if you it would ignore it completely? Wouldnt break the command as well?
I'd also tolerate skipping the test if the built binary doesn't exist
I think that is a great solution. I think in I had initially suggested something like that. If everyone is ok with that, I will go ahead and make that change.
Sounds good, make that change and we'll see how it turns out. I'd really want these tests start working, so +1


Question: func (f *fakeExtender) Bind(binding *v1.Binding) error { if f.isBinder { <ins> if f.errBind { return errors.New("bind error") } </ins> f.gotBind = true return nil }

I use extenders to add webhook for default-scheduler: And set to ignore scheuler error when scheduler extender is not working. But I got an error: . I suppose work fine in predicate path, but not work fine in bind path. The Ignorable logic in kube-scheduler: 637: and, 905: the is not judge in . Schedule should not error when set . extenders to kube-scheduler config(which has filterVerb and bindVerb and ignorable flag), example: and add config flag in kube-scheduler pod start up command. a scheduler-extender() and then stop it. a pod and you will see the schedule error. No response <details</details<detailsLocal environment </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Maybe this is not a bug but as designed, because bind is more critical than filter, what would you do in bind? I have no history context, cc
Hi, Kerthcet, in my scene, I use extender to bind IP(from my own network system) to my pod's annotaion. I'm aware of the critical of binding path, but, what If the extender is broken, how to skip it and continue? The scheduler extender design has supported ignorable, but only work for filter path make no sense for extender both has filter and bind. Maybe we can add an option to allow user to chose what can be ignored. Thanks for your reply anyway~
Reiterating the mentions to trigger a notification: <detailsIn response to : Maybe this is not a bug but as designed, because bind is more critical than filter, what would you do in bind? I have no history context, cc I'm aware of the critical of binding path, but, what If the extender is broken, how to skip it and continue?

The scheduler extender design has supported ignorable, but only work for filter path make no scence for extender both has filter and bind. Maybe we can add an option to allow user to chose what can be ignored. thanks for your reply anyway~ Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
So even the extender bind failed, will not impact your pod scheduling, right?
Yup
Right, if the field is there, we should support it. It's up to the cluster administrator to judge whether it makes sense to ignore a failed extender during bind.


Question: if proxier.ipvsScheduler == "mh" { serv.Flags |= utilipvs.FlagSourceHash } <del> if err := proxier.syncService(svcPortNameString, serv, true, alreadyBoundAddrs); err == nil { </del> <ins> // We must not add the address to the dummy device if it exist on another interface shouldBind := !nodeAddressSet.Has(serv.Address.String()) if err := proxier.syncService(svcPortNameString, serv, shouldBind, alreadyBoundAddrs); err == nil { </ins> activeIPVSServices.Insert(serv.String()) <del> activeBindAddrs.Insert(serv.Address.String()) </del> <ins> if shouldBind { activeBindAddrs.Insert(serv.Address.String()) } </ins> if err := proxier.syncEndpoint(svcPortName, svcInfo.ExternalPolicyLocal(), serv); err != nil { klog.ErrorS(err, "Failed to sync endpoint for service", "servicePortName", svcPortName, "virtualServer", serv) }

Hey, after upgrading kubernetes from 1.22 to 1.23, I'm experiencing strange service provisioning behavior using nodePort. I'm using kube-proxy (ipvs) + calico + metallb. If external IP is used by another service type (Loadbalancer - allocated by metallb), then the node owning this IP doesn't listen to NodePort on this interface. Unfortunately, I can't find any changes that could be causing this, any ideas? Kubernetes 1.23 Missing externalip1 here: Using k8s 1.22 service with nodeport caused listening on every interface, despite using external IP by another service. Kubernetes 1.22 Upgrade k8s from 1.22.15 to 1.23.15 No response <details</details<detailson-premise instances </details<details</details<detailskubeadm </details<detailsdocker </details<detailscalico v3.24.5 metallb v0.13.7 ipset v7.5, protocol version: 7 ipvsadm v1.31 2019/12/24 (compiled with popt and IPVS v1.2.1) #ipvs config: </details
/sig network
/assign
From : This is not a valid setup. A virtual IP (VIP), like the loadBalancerIP, must not be assigned to a physical interface. The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without_ assigning the VIP to any interface.
When you send traffic to a nodePort, the destination address should be a node address, not a loadBalancerIP.
There is however a change in the implementation (I will try to find the PR) in the way the nodePort addresses are computed. This is how i should be; On all clusterIP, loadBalancerIP and externalIPs are assigned. So the current implementation is correct. What I suspect that pre v1.23 K8s did this wrong by adding all addresses on interfaces except kube-ipvs0 (including loopback which would not work).

The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without assigning the VIP to any interface. These loadBalancerIP addresses are just the external IP of my nodes, which I use in metallb as an available pool for service(loadbalancer). Provider automatically assigns this address to the physical interface as a lease, I don't really have the option of a different setup at the moment. Ye, I know. In my case the public IP address pool for node and loadBalancerIP addresses are the same
So, the loadBalancerIPs assigned by metallb is actually the real addresses of your K8s nodes?
Yes, these are public IP addresses (each node has its own), which I use as an entry point for some services(loadbalancer) but I also have an external loadbalancer that targets nodePort. To eliminate SPoF, it directs traffic to the service (NodePort), which has been listening on all nodes so far, regardless of IP assignment to another service
Is this setup supported? Thanks for explaining. I understand your setup now, . I am unsure if it's supposed to work, so I have to ask. Have you tested proxy-mode=iptables? It may work, but I wouldn't be suprised if it doesn't.
I will try to test it with iptables, but I use ipvs because of specific services, so I can use the load balancing algorithm - least connection... this method won't solve my problem :(
Found the PR
Why do you need nodePorts services at all? Can't you use only loadBalancer? NodePort is mainly intended to be used by and external load-balancer, i.e. like in AWS, but when you manage your own cluster you usually don't have an external loadBalancer so use only loadBalancerIPs makes sense. We do that, and even .
Scratch that! In your case where the node addresses are the external addresses you should use NodePort only. If you use your node addresses as loadBalancerIP you will hit this PR when you upgrade to later K8s versions.

It inserts a blocking rule in the INPUT chain for loadBalancerIPs. This will block input to your node if loadBalancerIP==nodeIP.
I understand, thanks for the info. In my case it's a breaking change and I will have to migrate some services to NodePort before kubernetes upgrade on prod environment.
with proxy-mode=iptables, everything started working, node is listening on nodePort and I can get to loadBalancerIP, I cleared the rules on each node, just to be sure However this is not a solution for me, I need the lc algorithm from ipvs
Port used in the LoadBakancer service example is within the default NodePort range. I think it can be converted to a NodePort service without affecting external users. Or is just an example? Or are there other reasons for type:LoadBalancer?
/area ipvs
This is just an example, I won't have much trouble with the migration. At most, I will divide the available lease of IP addresses into smaller pools, for NodePort / Loadblancer services. I think I already know everything and there should be no more surprises. Thanks for the information and your time!
It is a little "unusal" to use the host IP as the LB IP, but it's unfortunate that we broke something that used to work. Lars, is there any path back to "working" that doesn't lose the protections in that linked PR? I could easily see iptables also breaking this..
To not exclude loadBalancerIPs (and ecternalIPs) when they are on a physical interface is not hard, but messes up the nice set-operation. The access protection in It should basically not be there if the any loadBalancerIPs (and ecternalIPs) is also on aphysical interface. Can be done, but may be messy.
/triage accepted Even though this issue is accepted, it's still under investigation. I will check the code if the old bevaivor can be restored in a fairly simple way. But there is a problem with back-port. If I make a PR now it will be in v1.26.x or v1.27 but the function was broken in v1.23 so a fix should really be back-ported all the way back.

I would prefer to document that a real address on an interface on the node can't be used as a virtual address, like loadBalancerIP or externalIP.
Just wanted to chime in and say that I've also encountered this issue... very grateful to find the discussion and looking forward to a solution. Thank you all! Also just to share, I've tried the following 3 workarounds and they all work in my case: Switching to iptables mode. It works I think because the iptables NAT rule refers to the hostname as the destination, which gets resolved to the public node IP. Disable IPv6 on my OS. This works too as it makes the code fall back to the single stack mode, which grabs the node IP also from looking up the hostname. Explicitly configure the field in the kube proxy config, that overrides the default get local addresses.
I backed down to K8s v1.22.4 and I can't get it to work unless you have the master outside the cluster (i.e. not a node), or have a setup where the external-ip you set as loadBalancerIP is not on the main K8s network, something like; <img src="" width="65%" /K8s is using the "backend" network, the "external" ips are on the "frontend" network. The reason is when you assign a loadBalancerIP, that address will be assigned to the interface, even before v1.23. So, on an in-cluster master node you will see something like; The address is the loadBalancerIP and is also the node-address of another node in the cluster, "vm-002". When the master receives "normal" packets from "vm-002" they will have src=192.168.1.2 but since that address is assigned to a local interface on the master the packets will be discarded as or spoofed packets. The result is that "vm-002" (e.g. on that node) loses contact with the master;
I discovered this when testing PR I think I can restore the pre v1.23 behavior now, but I suffer from the problem described above.
I don't think your problem is precisely the same. To disable ipv6 or set nodePortAddresses can't really be used as work-arounds for this issue.

Thank you, You're right about IPv6 - disabling it didn't really help. I forgot to remove the I put in while testing with disabling IPv6. It was the that makes the difference. That said, it really appears to work if we explicitly put in the . Maybe I'm still missing something in the code, but if I read it correctly, here is where we take out the load balancer IP: It is called from the folllowing, but only if contains any zero CIDRs. If a node port address is explicitly provided and it is NOT a zero CIDR, then will not be called. We're planning to roll out this workaround before upgrading to a future Kubernetes release with your fix. We would much appreciate it if you could shed more light why it wouldn't work. We didn't want to change to iptables mode if this second workaround really works. Thank you so much!
You are right. To explicitly set in the kube-proxy config works as a work-around. Still the loadBalancerIP (which is also an address on a node) can't be used by the main k8s network, or the master must not be a node in the cluster as described in And... This will work up to and including K8s v1.25.x, but NOT v1.26.0! I haven't checked yet but I think it's because of
Since the loadBalancerIP that is owned by a node is assigned to on other nodes you must stop those other nodes from responding to ARP. For example with:
From my perspective, this is not a big , I have the whole architecture implemented in separate networks. I think it's worth adding this information somewhere in the documentation.
Thank you so much for the response, epesically the info about v1.26. We'll check it out for sure and will report back. Our node port use case is for single-node deployments only, so luckily there are no other nodes.