Question: return nil }, nodeStatusUpdateTimeout, pollInterval).Should(gomega.Succeed()) }) <del> ginkgo.It("after restart dbus, should be able to gracefully shutdown", func(ctx context.Context) { // allows manual restart of dbus to work in Ubuntu. err := overlayDbusConfig() framework.ExpectNoError(err) defer func() { err := restoreDbusConfig() framework.ExpectNoError(err) }() ginkgo.By("Restart Dbus") err = restartDbus() framework.ExpectNoError(err) gomega.Eventually(ctx, func(ctx context.Context) error { // re-send the shutdown signal in case the dbus restart is not done ginkgo.By("Emitting Shutdown signal") err = emitSignalPrepareForShutdown(true) if err != nil { return err } isReady := getNodeReadyStatus(ctx, f) if isReady { return fmt.Errorf("node did not become shutdown as expected") } return nil }, nodeStatusUpdateTimeout, pollInterval).Should(gomega.Succeed()) }) </del> }) ginkgo.Context("when gracefully shutting down with Pod priority", func() {

Any test that runs Gracefulshutdown seems to be having issues. Reference: In this case, they are failing consistently. The tests were flaky but as of 9/29, they have been failing ever since. and I were looking into these failures. We are able to reproduce on a GCP instance but we are seeing failures to sending dbus shutdown signals. We have looked into this code but it does not look like any code is responsible for this failure. One thing we did see is that there was a new release of COS around then. We are seeing failures in the logs. Running this test locally we can sometimes see issues with dbus. Running this locally (after copying the image config): /sig node
/cc
/cc
/triage accepted /priority important-soon
Looking in the logs of the last successful job: And viewing the kernel logs, I don't see any errors related to dbus. Last successful job linux version: The next job fails and the main difference seems to be a linux kernel update.
/cc
/cc
I was able to repro this on COS M109 as follows, while the below works fine on COS M105. Will dig a bit deeper if this is an issue in COS or systemd upstream regression.
thank you for looking into this! (adding who authored the Dbus restart code) What we are seeing here would be something other than a regression. This problem affects Google's COS, Fedora CoreOS (and its RHEL-based equivalent), Fedora Workstations, and most likely any other Linux distribution, whether container optimised or not, that is not Debian or based on Debian, such as Ubuntu. Both Debian and Ubuntu are unaffected as they handle Dbus restarts slightly differently. TL;DR Restarting the Dbus broker service at runtime is generally not recommended or supported. Many systemd components (but also other software and services) that rely on Dbus wouldn't reconnect to a new Unix socket, which would have the same path (location) as before, following the Dbus broker restart. This can lead to anything from authentication timeouts (since the systemd-logind and the polkitd services would be impacted) to the system generally broken following Dbus broker restart - this is especially the case for desktop (such as Fedora Workstation) systems where there are a lot of variety of services that rely on working Dbus connectivity.

Different software might handle this differently and choose to reconnect on a socket write or read timeout. However, systemd at large is not one of these. Lennart has confirmed this on a number of occasions per (selected from a sample of issues): Debian and its derivatives, such as Ubuntu or Linux Mint, etc., are not affected as they opt to trigger the restart (or activation, if you wish) of key system services (such as the systemd-logind) following a Dbus broker restart. I recommend removing the Dbus broker restart from the test code base to fix the failing tests. If there were no objections, then I would be happy to do this work. Additionally, we could see about porting Debian's behaviour to Google's COS and Fedora CoreOS (and, as such, the Red Hat RHEL-based equivalent) around restarting systemd-logind on Dbus restart. But that would be a bonus work and outside of this issue's scope. Typically on a Fedora CoreOS distribution, the default list of Dbus connections just after system startup might look as follows: <details<summaryDbus connections list on Fedora CoreOS after startup</summary</detailsSeveral key systemd services are connected, such as the systemd-logind and the systemd-homed, plus the polkitd service. Restarting Dbus would reduce this list to the following: <details<summaryDbus connections list on Fedora CoreOS after Dbus broker restart</summary</detailsThere is almost nothing connected following the restart. Any attempt to introspect the /org/freedesktop/login1 object would now fail per: <details<summaryObject introspection run</summary</detailsRestarting the systemd-logind service causes a fresh process to reconnect to the Dbus bus per: <details<summaryDbus connections list on Fedora CoreOS after systemd-logind restart</summary</detailsThis resolves the problem of accessing the /org/freedesktop/login1 object per: <details<summaryObject introspection run</summary</detailsHowever, other services might still be affected and would need to be restarted. I mentioned that Debian and its derivatives are not affected as they trigger the restart of key systemd services, especially the systemd-logind service.

This can be seen in the following output - pay close attention to the PID of the systemd-logind service on the list (using Debian 12 "bookworm" as an example): <details<summaryDbus connections list on Debian 12 following Dbus broker restart</summary</detailsNote that for this particular Debian 12 system, the initial state of the connections was different. The following is the state immediately after the system starts up, this is a desktop flavour with a lot of other software that relies on Dbus, and that will also fail to reconnect correctly per: <details<summaryDbus connections list on Debian 12 after startup</summary</detailsWhen the Dbus broker is restarted, and a service that exposes an object hasn't reconnected to the new Unix socket, the following breakdown in connection between the client, the Dbus bus, and the service of interest in question can be seen. Any attempt to access the /org/freedesktop/login1 object will not work: <details<summaryObject introspection output</summary</detailsThis failure can also be seen on a bit lower level too, per: <details<summaryThe busctl strace output</summary</detailsA test has been in the past to catch issues following the Dbus broker service restart as part of a genuine issue that consisted of adding support for reconnecting to the Dbus socket in case of errors or connection failure. This test would work fine on Debian and Ubuntu, as these Linux distributions aren't affected as much as we already learned. Specific commit: This was in response to a reported issue: Given that tests are nowadays commonly run on either Googe Container-optimised OS (COS) or Fedora CoreOS, the addition of the Dbus broker restart would have side effects: Any use of would either appear as hang or take a long time to time out Attempt to sign into a given compute instance using SSH would have taken a long time or appear as a hang connection Dispatching requests to Dbus to send a signal to some other participating services can also take a long time before eventually timing out Some combination of the above might help explain why a lot of our tests were timing out either on connection (SSH) or execution (Ginko, etc.).

Related code: The code responsible for sending a signal (SIGHUP) to systemd-logind to ask it to reload its configuration: The above is an interesting take on sending the SIGHUP signal to a process since we don't need to find out what PID the process in question has and just ask Dbus to do the work for us - a bit unorthodox and relies on the service that when breaks would render other interactions with the system and other services unreliable. Related upstream issues and Pull Requests for reference: Linux distributions used for testing: <details<summaryA list of Linux distributions showing the /etc/os-release file content</summary</details
I took the time to diagnose it this week, and I tried to find a way around it. The previous purpose was to fix that kubelet could not work after dbus restart. Currently, this issue has been fixed in kubelet for a long time, and I have no opinion on deleting this test.
Hi This is working without issues. The issue is not between kubelet and Dbus, but between Dbus and other services that do not handle the Dbus broker being restarted. We are good on the kubelet front. :smile:


Question: func (ss *scaleSet) deleteCacheForNode(nodeName string) error { cached, err := ss.vmssVMCache.Get(vmssVirtualMachinesKey, cacheReadTypeUnsafe) if err != nil { <ins> klog.Errorf("deleteCacheForNode(%s) failed with error: %v", nodeName, err) </ins> return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: vmss dirty cache issue, caused by What you expected to happen: clean vmss cache should happen after disk attach/detach operation, now it's before those operations, which would lead to dirty cache. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: } } <del> // Delete the PV. Fail test if delete fails. </del> <ins> // Delete the PV. Fail test if delete fails. If success the returned PV should // be nil, which prevents the AfterEach from attempting to delete it. </ins> func deletePersistentVolume(c *client.Client, pv *api.PersistentVolume) (*api.PersistentVolume, error) { if pv == nil { return nil, fmt.Errorf("PV to be deleted is nil") } <del> By("Deleting PersistentVolume") </del> framework.Logf("Deleting PersistentVolume %v", pv.Name) err := c.PersistentVolumes().Delete(pv.Name) if err != nil {

Context: NodeSwap feature gate was recently . Currently, as per , swap is only supported with cgroup v2 - cgroup v1 is no longer supported. For this reason, we decided to fail kubelet if NodeSwap feature gate is enabled and the node runs with cgroup v1. This would help prevent any surprises for the user by failing as soon as possible. This effort is being implemented in this PR: The problem: Currently, swap jobs are running with cgroup v1. Swap tests were in a way that they first check on which environment they're being run (e.g. which cgroup version? is NodeSwap FG enabled? Which swapBehavior is defined? etc) and then expect the right behavior to take place. With v1, the tests expect the containers to be defined without swap access. While the tests pass, they are much less valuable, since all they check is that swap is disabled for v1 workloads. The tests would be much more valuable if they would run with v2, therefore actually ensure that proper swap limitations are configured. In addition, is dependent on these changes, since as of now the jobs immediately fail with this PR since kubelet fails on an environment with cgroup v1 and swap enabled. What needs to change: [x] CRI-O swap related jobs should be defined to run with cgroup v2 (e.g. ) [x] containerd swap related jobs should be defined to run with cgroup v2 (e.g. ). Future work: In the future, we might consider introducing more jobs to check swap. Ideally, both and would be tested, perhaps by periodic jobs.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/sig testing
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: s.Lock() defer s.Unlock() for i := begin; i <= end; i++ { <del> s.used.SetBit(&s.used, i, 1) s.allocatedCIDRs++ cidrSetAllocations.WithLabelValues(s.label).Inc() </del> <ins> // Only change the counters if we change the bit to prevent // double counting. if s.used.Bit(i) == 0 { s.used.SetBit(&s.used, i, 1) s.allocatedCIDRs++ cidrSetAllocations.WithLabelValues(s.label).Inc() } </ins> } cidrSetUsage.WithLabelValues(s.label).Set(float64(s.allocatedCIDRs) / float64(s.maxCIDRs))

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With a /22 pod network and a /24 assigned to each node, a third node cannot get a pod CIDR if kube-controller-manager has ever been restarted. What you expected to happen: Once a third or fourth node joins, it gets a /24. How to reproduce it (as minimally and precisely as possible): 150% usage on the IPAM controller is not a good sign. Anything else we need to know?: During the creation of the rangeAllocator, Occupy() is called twice on the pod networks. The first time is before subscribing to the node informer. The second time is during the node informer callbacks because on subscribing it calls Add for all existing nodes. I will submit a proposed fix shortly. Environment: Kubernetes version (use ): 1.19.2
/sig network
/triage accepted thanks for finding the bug I 've introduced :)


Question: apiVersion: v1 kind: Service metadata: <del> # This service is meant to be used by clients of the database. It exposes a ClusterIP that will # automatically load balance connections to the different database pods. name: cockroachdb-public labels: app: cockroachdb spec: ports: # The main port, served by gRPC, serves Postgres-flavor SQL, internode # traffic and the cli. - port: 26257 targetPort: 26257 name: grpc # The secondary port serves the UI as well as health and debug endpoints. - port: 8080 targetPort: 8080 name: http selector: app: cockroachdb --- apiVersion: v1 kind: Service metadata: </del> # This service only exists to create DNS entries for each pod in the stateful # set such that they can resolve each other's IP addresses. It does not # create a load-balanced ClusterIP and should not be used directly by clients

Seems to be broken for at least the past two weeks. /sig apps /area test
can you take a look please?
Absolutely, I'll take a look today. Sorry guys!
thanks!


Question: <ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "fmt" "k8s.io/kubernetes/pkg/api/meta" pkg_runtime "k8s.io/kubernetes/pkg/runtime" ) const ( //TODO: This will be removed once cluster name field is added to ObjectMeta. ClusterNameAnnotation = "federation.io/name" ) // TODO: This will be refactored once cluster name field is added to ObjectMeta. func GetClusterName(obj pkg_runtime.Object) (string, error) { accessor, err := meta.Accessor(obj) if err != nil { return "", err } annotations := accessor.GetAnnotations() if annotations != nil { if value, found := annotations[ClusterNameAnnotation]; found { return value, nil } } return "", fmt.Errorf("Cluster information not available") } // TODO: This will be removed once cluster name field is added to ObjectMeta. func SetClusterName(obj pkg_runtime.Object, clusterName string) error { accessor, err := meta.Accessor(obj) if err != nil { return err } annotations := accessor.GetAnnotations() if annotations == nil { annotations = make(map[string]string) accessor.SetAnnotations(annotations) } annotations[ClusterNameAnnotation] = clusterName return nil } </ins>

Federated controllers will oftentimes have to crreate/update/delete the same/similar structures in multiple clusters. Given that the roundtrip time to remote clusters can be high (like, lets say, 1 sec) these operations should be done in parallel so that the main controller goroutine is not blocked for long. Part of


Question: <del> This file is autogenerated, but we've stopped checking such files into the repository to reduce the need for rebases. Please run hack/generate-docs.sh to populate this file. <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/user-guide/kubectl/kubectl_edit_configmap.md?pixel)]() <!-- END MUNGE: GENERATED_ANALYTICS --> </del>

kubectl edit is not working on HEAD. It's working on 1.5.


Question: var gracePeriod int64 = 1 var command string <ins> /** This condition fixes running storage e2e tests in SELinux environment. HostPath Volume Plugin creates a directory within /tmp on host machine, to be mounted as volume. Inject-pod writes content to the volume, and a client-pod tries the read the contents and verify. When SELinux is enabled on the host, client-pod can not read the content, with permission denied. Invoking client-pod as privileged, so that it can access the volume content, even when SELinux is enabled on the host. */ if config.Prefix == "hostpathsymlink" || config.Prefix == "hostpath" { privileged = true } </ins> command = "while true ; do sleep 2; done " seLinuxOptions := &v1.SELinuxOptions{Level: "s0:c0,c1"} clientPod := &v1.Pod{

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Some of our storage e2e tests write and read data in the host FS (/tmp), using the hostPath storage plugin. By default, containers running on SELinux environments can only read/write data in directories/files (in the host) that contain the label . In the absence of this label, there are two options: 1) Run the container as privileged or; 2) In the host, add this label to the direcotories/files in which the container will write data (and keep the container unprivileged). Currently, we run the container as privileged if the host is not Windows and the volume is FS (rather than block). That being said, we should run containers as unprivileged in all cases (e.g., option 2 above), so I propose to: 1) Prepare a special e2e directory in the host by adding the proper label to it (); 2) Change the tests to read/write data from/to this directory instead (they typically use /tmp/ for that); 3) Make the test containers run as unprivileged. Reference:
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this issue still open?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale yes, we'd like to have this fixed eventually.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten


Question: } limit2, ok := obj2.(*example.PodList) if !ok { <del> t.Fatalf("Expected PodList but got %t", limit2) </del> <ins> t.Fatalf("Expected PodList but got %v", limit2) </ins> } if limit2.Continue != "" { t.Errorf("Expected list not to have Continue, but got %s", limit1.Continue)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Unit tests are failing with the latest golang(1.12) What you expected to happen: Tests should pass without any fauilures. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig api-machinery


Question: } testItem := func(name string, item *Item) { <del> // Canonicalize the html so we can diff. item.input = fmtHTML(item.input) item.output = fmtHTML(item.output) </del> server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Check request headers. if got, want := r.Header.Get("X-Forwarded-Uri"), item.forwardedURI; got != want {

The ApiServer proxy wraps partial HTML responses to serve a complete HTML document in the proxy response. This is problematic for applications which expect HTML fragments, such as with AngularJS templates. Example: This is the response for a from the unproxied server: But the proxied response is: This issue is blocking


Question: // healthCheckNodePortUpdate handles HealthCheckNodePort allocation/release // and adjusts HealthCheckNodePort during service update if needed. <del> func (rs *REST) healthCheckNodePortUpdate(oldService, service *api.Service) (bool, error) { </del> <ins> func (rs *REST) healthCheckNodePortUpdate(oldService, service *api.Service, nodePortOp *portallocator.PortAllocationOperation) (bool, error) { </ins> neededHealthCheckNodePort := apiservice.NeedsHealthCheck(oldService) oldHealthCheckNodePort := oldService.Spec.HealthCheckNodePort

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: healthcheck nodeport allocation doesn't have roll-back, this may lead to nodeport leak. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig network /assign Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig network /assign
Could you clarity "roll-back"? It seems that we basically in the same way.
I didn't mean the HealthCheck NodePort release process in . I meant the allocation process in and . Take for example, please see: There is a defer for common NodePorts in : Once fails, the above defer will finally exec. Then the allocated NodePorts will roll-back, i.e. these NodePorts will be released. But the HealthCheck NodePort allocation process doesn't use this roll-back. This will lead to leak. I run a simple test and see the HealthCheck NodePort leak. Will paste the result later.
Testing service manifest, . some simple testing code. See: . Note that I add this error intentionally: kube-apiserver with flag . . We will get below output. This is expected. again. We will get below output. Note the error . That means that when creating service fails at step 3, the NodePort and HealthCheck NodePort should be both released, but the was NOT.
Fortunately, several minute later, the leak was repaired and cleaned up. But anyway, we should release the HealthCheck NodePort when fails. We shouldn't depend on the later repair.
Thanks for tracing down, sent for fixing.
We're having this issue on service deletion. It looks like healthcheck nodeport is not deallocated immediately when deleting the service. This is causing issues for backup and restore of services with heathcheck node port.


Question: if err != nil { return err } <ins> volPath = removeClusterFromVDiskPath(volPath) </ins> diskID, err := getVirtualDiskID(volPath, vmDevices, dc, vs.client) if err != nil { glog.Warningf("disk ID not found for %v ", volPath)

With version 1.4.7 I could use a datastore that is in a cluster as a vsphere volume as . When I upgraded to 1.6.0, this same exact path does not work and throws a error. I'm struggling to find what changed, but am guessing it's difference in how the datastore is being looked up in vcenter. Has anyone had luck with this?
We have verified creating pod using static PV and dynamic provisioning, both works fine with Can you post here? cc:
Did that include using a datastore that was in a datastore cluster? That seemed to be working before because the datastore previously (before 1.5) was read out of the global config, which supported datastore path with cluster name. But post 1.5 datastore is parsed from the entry, which if you include the path to the cluster it gets you past the datastore lookup, but then when you go to add the disk it fails because that is not valid to vCenter. It's kind of this weird mixed mode, because to look up the datastore you need the path including the cluster, but to create the file you need the path without the cluster. I presume the PV is working because they use the cloud config values? I was not using a PV but putting a volumePath directly in my deployment yaml. Should I be using a PV to do this? If the deployment is deleted and re-created how do I guarantee that my pod would get the same PV claim? That was why I did not go down that path originally. I'm pretty sure I have narrowed down where in the provider this is happening, but didn't see a good path to get it to work. I have, admittantly, not tried PV/PVCs yet, due to the concern raised above, but could try those to see. The pertinent part of the deployment yaml: The cloud config:
Just confirmed I have the same issue using a PV and PVC....
Here it needs it in format: (which it gets on line 791 by parsing . But here it needs it in true format, without the datastorecluster: Therefore if you do it gets you past line 796 but fails on 802. But if you do then you don't get past line 796.
Thanks for detailed look up. This is the commit which caused this regression. We will work on the fix.

cc:
Ah yes, I see that now. What is the lifecycle of a dynamic disk? Does that disk live on after the PVC is released? After the PV is deleted? It seems like I may just not fully be exercising the power of the volume plugins as well. I would still have the issue with existing volumes as they have data that I need to maintain, but would be interesting for new volumes to look into these more.
RECLAIMPOLICY for PV created using Dynamic Provisioning is Delete PV will get deleted if you delete PVC. Disk also gets deleted from datastore.


Question: stopCh := make(chan struct{}) s := NewStore(MetaNamespaceKeyFunc) listCallRVs := []string{} <ins> version := 30 </ins> lw := &testLW{ WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) {

What happened: Using client-go v1.18.8 to watch resources on API server versions 1.17.x and 1.18.0 - 1.18.5 After the connection to the server is temporarily interrupted the client never recovers from error See the earlier issue for steps to reproduce: This was assumed fixed by But it appears the fix for the client was (probably inadvertently) implemented in a non-backward-compatible fashion and only solves the issue when the server is running v1.18.6 or later as well.
/sig api-machinery
Cc
PR
We generally recommend having apiservers be updated first before all other components. But yeah - it's true that this doesn't work for clients being more fresh - let me take a look at your PR.
/cc
This will be addressed by (and the cherrypicking it back).
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This has been fixed by and corresponding cherrypicks.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I have disconnected a node from network for a few minutes. After reconnecting, I keep receiving such error messages from kubelet on the node, even after 15 minutes in reconnected state: What you expected to happen: I expect that after network recovery kubelet reconnects to the Apiserver as before, and after a recovery such timeouts do not occur. How to reproduce it (as minimally and precisely as possible): Just have a node connected to the cluster. Then, disconnect it from the network for 3-4 minutes, then reconnect. Then observe kubelet's logs. Anything else we need to know?: I have strict tcp keepalive settings in place on master and worker nodes, but this should not be the cause. Restarting kubelet solves the issue, the error messages disappear. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: bare metal OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): kube-router Others:
/sig api-machinery
/sig api-machinery
It seems that after some more time, kubelet recovers. Sometimes it takes one hour, sometimes two. Recovery is successful when I receive similar logs: What is the reason that it takes so long for it to recover?
The root cause is , deduplicating against that issue
Debugging, tracing the code, I am not really sure that this is the same issue as the other mentioned here. Error messages show that the client is requesting higher resource versions that is available at the server: , the former is greater by 989 than the latter. I dont know from where this resource version comes from. I've made a very stupid patch on my kubelet/client-go: I am pretty sure that this is not the best solution, but, it seems that it forces a full list every time ListAndWatch is called.
You're correct, sorry about that... I misread the timeout message. The error it is getting is coming from the server, which means kubelet -apiserver connectivity is fine.

This message comes from the watch cache, which means the watch cache in the API server is behind where the kubelet's informer was, which is pretty surprising. A few questions: Is there a single apiserver or multi-apiserver cluster? Is there a single etcd member or multiple etcd members? Is there an API server running on the node that was detached/reattached?
cc does this ring any bells?
cc for eyes on watch cache
I don't think I've hit this problem specifically. Since this could happen due to an etcd partition or due to a partition between a apiserver and etcd, the answers questions asked in seem like the appropriate next things to figure out.
Yeah - I agree with Joe. It seems like some network partition issue of the apiserver the node was connected to.
I will be able to answer the questions in detail later, but in short: there are 3 masters set up with kubeadm, haproxy load-balances the queries to the apiservers. The only thing to reproduce the issue is to disconnect the node from the network for a few minutes. Masters or etcd members are not touched, and anyway the cluster seems to be healthy.
Today, it occured on a diffrent cluster. Setup is similar: stacked etcd on master nodes, haproxy load-balances traffic between them. Just was playing with haproxy servers and the HA setup, and after a few restarts of haproxy, one of the nodes entered this state. So during a haproxy restart, all kubelets' connections to apiservers are terminated, but rarely, one of them enters this unhealthy state.
Multi apiserver cluster Stacked etcd members are present. I think In my setup it is not relevant, as all the kubelets are talking to the apiserver through the load-balancer, which picks one of the available apiservers. I am doing a rolling restarts of the nodes, and there were at least one occurence, when even one of the master nodes got stucked in this state. How can I help debugging/fixing this?
How can we proceed with this issue?

got same problem in a old cluster, ( up since 3+years) after updating to kube 1.18.3 , one node kubelet start logging those errors ( ~10 errors/minute with various resources) I've now just restarted kubelet and now seems working again.. Update: while testing I've tried to connect the kubelet of that node to each single master nodes to check if there were some kind of network issue / partitioning.. none of the master servers returned errors ( the kubelet worked fine with all master nodes bypassing the nginx proxy that manage masters HA
I think this problem is getting serious. I am not 100% sure, but today, during node restarts (including masters) coredns was migrated multiple times, and unfortunately, kube-proxy pods had not received the updated situation, thus, dns resolution was not working inside the cluster. After detecting this, a simple restart of all kube-proxy pods resolved our problem. There were no network issues during this, so I suspect we've hit the same bug, now in kube-proxy.
I am trying to dig this. Now, I've written a little go prog which uses the same reflector what kubelet uses. It just starts a watch for CSIDrivers, and I've made client-go to print the last resourceVersion. Starting the program multiple times (i.e. connecting to different masters) produces the following: Right now I dont exactly know what url is being fetched and what arguments are passed in, I still have to figure that out. But now it seems, that different masters return different resouceVersions. Etcd does not report any problems/issues.
Any ideas, suggestions where to look further?
Today I have re-initialized 2 of my 3 master nodes to make sure etcd replicas are not corrupted. Just after the nodes have been joined, the same behavior could be observed: connecting to different masters resulted in different resourceVersions.
Now, it can be seen, that querying different apiservers again return different resourceVersions: 192.168.8.60: is a haproxy load-balancing traffic between 3 masters.
please let me know where to go further. My cluster runs on PI boards, perhaps, not the fastest ones.

But if this issue depends on slow hardware, than it can pop up anytime on fast hardware as well.As I wrote, today I reinitialized 2 of my 3 masters to make sure no partitioning occured. Also, please have a look at my comments. Or, if you have a clue that my setup is broken, please let me know!
It seems that it happens to resources where no objects exist at all. Can it be?
Ahh - the fact tht it happens for types with no objects actually made me know what is happening here. Basically what is happening is that when kube-apiserver is starting, watchcache is initialized from "now" (i.e. it will be synced to resource version that was globally last in etcd at the time when it was initialized). However, later it's updated only on changes of resources of those types. So when no objects of that type is being /modified/deleted it will stay the same. So it's kind-of "working-as-implemented" from the kube-apiserver perspective. That can be improved - I'm actually already looking into reusing etcd progress-notify feature to get rid of this behavior ( & ) - I would like to have alpha for that in 1.20 and beta in 1.21 (planning to write down KEP once I'm back from my soon-starting vacation), but that shouldn't be the problem by itself. And it's not what is really biting you. The real problem is that there is a bug in reflector in: It handled the "too old RV" error, but it doesn't handle "too large RV" correctly. What it should be doing instead is to: trigger relist from etcd (to get RV=now") start watching from that RV Watch will work fine because watch will simply wait until events with larger RV appear. I will send out a PR fixing that bug (will try to do it later today).
is out for review
This has been fixed/mitigated in 1.18. Going to backport to 1.18 now. It seems 1.18 is the first version affected (the offending changes were reverted in 1.17 months ago).
This has been mitigated in 1.18 and head. The more proper fix is proposed in Closing this one.

Hello, as i saw, i have the same issue in 1.19.3 version: Our setup is 3 etcd, 2 apiserver bound on those 3 etcd and a haproxy in front of apiservers
If that happens once - that can definitely happen (and that's fine). The bug was that the components were stuck with those errors.
18.0 get same error . restart docker and kubelet return OK. so , should I update the 1.18.0 to 1.18.6+ ?


Question: return p } <ins> func getPodResourcesMetrics() (e2emetrics.KubeletMetrics, error) { // we are running out of good names, so we need to be unnecessarily specific to avoid clashes ginkgo.By("getting Pod Resources metrics from the metrics API") return e2emetrics.GrabKubeletMetricsWithoutProxy(framework.TestContext.NodeName+":10255", "/metrics") } func timelessSampleAtLeast(lower interface{}) types.GomegaMatcher { return gstruct.PointTo(gstruct.MatchAllFields(gstruct.Fields{ // We already check Metric when matching the Id "Metric":    gstruct.Ignore(), "Value":     gomega.BeNumerically(">=", lower), "Timestamp": gstruct.Ignore(), })) } </ins>

In commit we podresources-related metrics, to make sure the feature was monitoring-friendly. Because of a implementation bug, the metrics have been , they are updated correctly, but they are never registered, so they don't show up when doing like the podresources related metrics should be reported in the endpoint in the kubelet the kubelet, even the simplistic environment of the tests is sufficient sure to enable the and feature gates (on by default in kube = 1.25 at least) the podresources endpoint issuing and calls. The actual output is not relevant the metrics: The bug was introduced in the original commit <details</details<detailsNot relevant. </details<detailsNot relevant </details<detailsNot relevant </details<detailsNot relevant </details<detailsNot relevant </details
/sig node /assign
/cc /triage-accepted
/triage accepted /cc


Question: {utiliptables.TableNAT, KubeNodePortChain}, {utiliptables.TableNAT, KubeLoadBalancerChain}, {utiliptables.TableFilter, KubeForwardChain}, <ins> {utiliptables.TableFilter, KubeNodePortChain}, </ins> } // ipsetInfo is all ipset we needed in ipvs proxier

Before switching the modes from to it makes sense to cleanup unnecessary rules. This can be done using the command line option of . Unfortunately, this does not work anymore since was introduced into the table with We see the following error when trying to cleanup the rules after running in mode: Essentially, ipset could not be deleted as it was still referenced by the chain in the table. Cleanup of should not throw any error and completely remove the rules it created. with enabled. (You may add the configuration provided in the "What happened?" section, but it should fail either way.) The issue can be trivially fixed by adding to in . <details</details<detailsn/a </details<details</details<detailsn/a </details<detailsn/a </details<detailsn/a </details
/sig network
it seems we forget to clean these new rules :smile:
/assign /triage accepted
I see, let me know if you need more context/assist!


Question: [Service] ExecStartPre=-/usr/bin/mkdir -p /opt/bin <del> ExecStartPre=/usr/bin/wget -N -P /opt/bin https://storage.googleapis.com/k8s/setup-network-environment </del> <ins> ExecStartPre=/usr/bin/wget -N -P /opt/bin https://github.com/kelseyhightower/setup-network-environment/releases/download/v1.0.0/setup-network-environment </ins> ExecStartPre=/usr/bin/chmod +x /opt/bin/setup-network-environment ExecStart=/opt/bin/setup-network-environment RemainAfterExit=yes

This is missing and will break all CoreOS related stuff . /cc
looks so - i just did AntonioMeireles/kubernetes-vagrant-coreos-cluster to get around it ... waiting a bit to see if it was transient or not before propagating it urbi et orbi.
Well, I think it's better if we just use the upstream and not said missing URL.
This is not resolved by updating to the upstream URL for the setup-network-environment binary. I've turned over the k8s bucket on GCE so I don't plan to use it going forward.
I'm not sure I follow you So you mean this binary is no longer needed? I don't understand why the issue was closed.
Nevermind


Question: gcloud docker -- push ${REGISTRY}/hyperkube:${VERSION} endif <del> .PHONY: all </del> <ins> .PHONY: build push all </ins>

I'm pretty sure that building the hyperkube images takes longer than building kubernetes itself now. At least a large part of this seems to be that we download and install ~172 extra debian packages several times (once for each arch). <details</detailsNot sure how to improve this - maybe better caching? Do we actually want to install these extra packages? Should we use a better base image than ? cc
cc (not really sure who "owns" hyperkube, if anyone)
The hyperkube binary is one thing. I think the image ends up being big as we drag in so much stuff. I wasn't involved in creating the docker image so I can't comment. Ideally this thing would be slim and would have minimal external dependencies. Fixing this starts with mapping why we have each of those deps imported.
I actually have written a such patch in my local env, but I never uploaded. It removed the docker and docker-multinode-specific stuff and all the unused deps. Also, we will come one step closer once is in. Should I submit the patch so you can take a look at it?
sure!
This needs to be triaged as a release-blocker or not for 1.5
punting to v1.6.
The main problem here is that we're installing huge packages for supporting some volume types: git nfs-common glusterfs-client cifs-utils With containerized volume mounters we'll get rid of that.
When it comes to local rebuilding, should help here. As I've already noted in the PR, maybe we should make CACHEBUST=0 the default for all hack/dev- scripts. This would at least ease development and testing.
can this be closed? or moved to 1.7?
should this be in v1.7? is there an appropriate sig to own this?
cries nope, not v1.7. I'm still not sure who owns hyperkube.
Everyone and no one I suppose. Yeah, agree it's sad.
/sig release guess we're saying sig-release owns hyperkube under the general guise of "cutting bits" until/unless there is a better home for it
Yeah, hyperkube definitely belongs to sig-release as it hasn't been owned by a sig so far...


Question: <ins> # See the OWNERS docs at https://go.k8s.io/owners approvers: - sig-scheduling-maintainers reviewers: - sig-autoscaling-maintainers - x13n </ins>

Unit test that will ensure that Cluster Autoscaler objects that currently implement interfaces from stay that way. It would suffice to do add compile-time checks for the following types: Interface changes could still be introduced in a PR against k/k repo, but they would require Cluster Autoscaler changes first to be merged before the interface change itself is merged. To avoid circular dependency issues, the unit test itself would have to live in a separate module inside k/k repo, importing both scheduler interfaces and CA implementations. Scheduler framework is an API used for communication between Cluster Autoscaler and Scheduler code. Since it is not a versioned API, it is easy to introduce changes that are backwards incompatible, which is exactly what happened recently in Ensuring interface compatibility in core k8s repo will prevent accidentally breaking Cluster Autoscaler code.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/sig autoscaling
I'm +1 and based on the discussion on last sig-autoscaling meeting the idea has a wide support in the sig-autoscaling. One question: shouldn't we also cover ?
Good point, CA is not only implementing scheduler interfaces, but also calling them, I think it makes sense to extend the scope to include in a compile-time check.
+1 I think it's not only for CA. But also for the plugins (not in k/k) implemented by users themselves. Ensuring interface compatibility is needed.
I understand the intention to prevent CA's code to be broken, however, the "CA's change first" strategy doesn't sound that practical as (1) that would require k/k to vendor CA's code, and (2) the "preventive changes" in CA doesn't really get verified at runtime/compilation time as the upstream changes are not merged yet. Given no matter what kind of changes go first, the scheduler framework changes need to get CA team involved. I'd suggest we continue to follow this rule, which works well all the time and is just violated once in .

In addition to the early involvement in the k/k PR's review, we can setup additional CI check in CA's side, so when the deps in CA, the CI should have (which isn't today).
The scheduler framework changes are somewhat unavoidable to evolve over time, aimed to behave performant as well as fitting more scenarios. The changes vary from minor changes like PodInfo-QueuedPodInfo, to major changes like interface addition/deletion/update, and hence those would require the consumers (like plugins) to adjust when bumping versions. I don't think we can promise strict compatibility at this moment.
Good point that we wouldn't be able to verify the code on CA side does indeed implement the updated interface. That might be sufficient, I'm just worried that this is not an automated process and hence can be skipped again. Is there anything we could do to limit the possibility of human error? My best idea here so far is to add sig-autoscaling as reviewers for framework changes. Or perhaps a subset of sig-autoscaling members that are cluster-autoscaler maintainers (as opposed to, for example, VPA). Actually, tests are already run for CA PRs, so it should already be detectable that way. The problem is that by the point it happens, the interface changes are already merged and there is no way to influence them to be more compatible with CA use case.
I think this is where unittests could help. The unittest wouldn't mean the interface cannot be changed, it would just provide a signal that sig-autoscaling / Cluster Autoscaler reviewer should be involved. It seems much more reliable than the current system based on scheduler contributors' intuitions on which changes may affect CA.
+1 on unit tests owned by sig-autoscaling
Unit test will block submission, which is more than just letting sig-autoscaling know they should take a look. In that case an integration test would probably make more sense: use GitHub API to pull latest version of kubernetes/autoscaler repo, then use Go compiler libs to try to compile selected files. On breaking interface changes, SIG autoscaling approver would be required to temporarily disable the test and drive adjustments on Cluster Autoscaler side (asynchronously, not blocking development on k/k repo). Thoughts?
I think we just need to have tests in k/k that hardcode the CA expectations rather than trying to think how to import CA.

Do you mean a slim fake library to simulate CA and other consumers? If so, it may still have gaps with CA. While if we import CA in k/k, again, that may yield a cyclic dependency. That sounds fair to me as it simulates how a consumer consumes the framework. Just not sure how complicated it can be (supposing we don't have similar tests to leverage).
I am hoping that we don't need to "simulate" CA; just their expectations of the framework itself.
I like suggestion: Simple test that validates CA expectations of the framework interface, without taking on all the extra complexity stemming from dependencies. This should be sufficient for now and may be replaced by more sophisticated mechanism I described above if needed. At the very least, it should suffice to create an object and check if it satisfies the framework interface. We can iterate on that later, if needed.
+1 with the suggestion. What about verify the framework at startup like , and make CA team part of the folder reviewer.
Yup, exactly something like this. I think putting it inside a unit test makes more sense though - there's no reason for this code to end up in scheduler binary.
would you like to finish that PR?
Sure, quite busy in the past few days. I'll complete this ASAP. Thanks for reminding.
Thanks, I left a minor comment on the PR, otherwise LGTM.


Question: "net" "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/klog/v2" </ins> utilproxy "k8s.io/kubernetes/pkg/proxy/util" netutils "k8s.io/utils/net"

Hey, after upgrading kubernetes from 1.22 to 1.23, I'm experiencing strange service provisioning behavior using nodePort. I'm using kube-proxy (ipvs) + calico + metallb. If external IP is used by another service type (Loadbalancer - allocated by metallb), then the node owning this IP doesn't listen to NodePort on this interface. Unfortunately, I can't find any changes that could be causing this, any ideas? Kubernetes 1.23 Missing externalip1 here: Using k8s 1.22 service with nodeport caused listening on every interface, despite using external IP by another service. Kubernetes 1.22 Upgrade k8s from 1.22.15 to 1.23.15 No response <details</details<detailson-premise instances </details<details</details<detailskubeadm </details<detailsdocker </details<detailscalico v3.24.5 metallb v0.13.7 ipset v7.5, protocol version: 7 ipvsadm v1.31 2019/12/24 (compiled with popt and IPVS v1.2.1) #ipvs config: </details
/sig network
/assign
From : This is not a valid setup. A virtual IP (VIP), like the loadBalancerIP, must not be assigned to a physical interface. The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without_ assigning the VIP to any interface.
When you send traffic to a nodePort, the destination address should be a node address, not a loadBalancerIP.
There is however a change in the implementation (I will try to find the PR) in the way the nodePort addresses are computed. This is how i should be; On all clusterIP, loadBalancerIP and externalIPs are assigned. So the current implementation is correct. What I suspect that pre v1.23 K8s did this wrong by adding all addresses on interfaces except kube-ipvs0 (including loopback which would not work).

The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without assigning the VIP to any interface. These loadBalancerIP addresses are just the external IP of my nodes, which I use in metallb as an available pool for service(loadbalancer). Provider automatically assigns this address to the physical interface as a lease, I don't really have the option of a different setup at the moment. Ye, I know. In my case the public IP address pool for node and loadBalancerIP addresses are the same
So, the loadBalancerIPs assigned by metallb is actually the real addresses of your K8s nodes?
Yes, these are public IP addresses (each node has its own), which I use as an entry point for some services(loadbalancer) but I also have an external loadbalancer that targets nodePort. To eliminate SPoF, it directs traffic to the service (NodePort), which has been listening on all nodes so far, regardless of IP assignment to another service
Is this setup supported? Thanks for explaining. I understand your setup now, . I am unsure if it's supposed to work, so I have to ask. Have you tested proxy-mode=iptables? It may work, but I wouldn't be suprised if it doesn't.
I will try to test it with iptables, but I use ipvs because of specific services, so I can use the load balancing algorithm - least connection... this method won't solve my problem :(
Found the PR
Why do you need nodePorts services at all? Can't you use only loadBalancer? NodePort is mainly intended to be used by and external load-balancer, i.e. like in AWS, but when you manage your own cluster you usually don't have an external loadBalancer so use only loadBalancerIPs makes sense. We do that, and even .
Scratch that! In your case where the node addresses are the external addresses you should use NodePort only. If you use your node addresses as loadBalancerIP you will hit this PR when you upgrade to later K8s versions.

It inserts a blocking rule in the INPUT chain for loadBalancerIPs. This will block input to your node if loadBalancerIP==nodeIP.
I understand, thanks for the info. In my case it's a breaking change and I will have to migrate some services to NodePort before kubernetes upgrade on prod environment.
with proxy-mode=iptables, everything started working, node is listening on nodePort and I can get to loadBalancerIP, I cleared the rules on each node, just to be sure However this is not a solution for me, I need the lc algorithm from ipvs
Port used in the LoadBakancer service example is within the default NodePort range. I think it can be converted to a NodePort service without affecting external users. Or is just an example? Or are there other reasons for type:LoadBalancer?
/area ipvs
This is just an example, I won't have much trouble with the migration. At most, I will divide the available lease of IP addresses into smaller pools, for NodePort / Loadblancer services. I think I already know everything and there should be no more surprises. Thanks for the information and your time!
It is a little "unusal" to use the host IP as the LB IP, but it's unfortunate that we broke something that used to work. Lars, is there any path back to "working" that doesn't lose the protections in that linked PR? I could easily see iptables also breaking this..
To not exclude loadBalancerIPs (and ecternalIPs) when they are on a physical interface is not hard, but messes up the nice set-operation. The access protection in It should basically not be there if the any loadBalancerIPs (and ecternalIPs) is also on aphysical interface. Can be done, but may be messy.
/triage accepted Even though this issue is accepted, it's still under investigation. I will check the code if the old bevaivor can be restored in a fairly simple way. But there is a problem with back-port. If I make a PR now it will be in v1.26.x or v1.27 but the function was broken in v1.23 so a fix should really be back-ported all the way back.

I would prefer to document that a real address on an interface on the node can't be used as a virtual address, like loadBalancerIP or externalIP.
Just wanted to chime in and say that I've also encountered this issue... very grateful to find the discussion and looking forward to a solution. Thank you all! Also just to share, I've tried the following 3 workarounds and they all work in my case: Switching to iptables mode. It works I think because the iptables NAT rule refers to the hostname as the destination, which gets resolved to the public node IP. Disable IPv6 on my OS. This works too as it makes the code fall back to the single stack mode, which grabs the node IP also from looking up the hostname. Explicitly configure the field in the kube proxy config, that overrides the default get local addresses.
I backed down to K8s v1.22.4 and I can't get it to work unless you have the master outside the cluster (i.e. not a node), or have a setup where the external-ip you set as loadBalancerIP is not on the main K8s network, something like; <img src="" width="65%" /K8s is using the "backend" network, the "external" ips are on the "frontend" network. The reason is when you assign a loadBalancerIP, that address will be assigned to the interface, even before v1.23. So, on an in-cluster master node you will see something like; The address is the loadBalancerIP and is also the node-address of another node in the cluster, "vm-002". When the master receives "normal" packets from "vm-002" they will have src=192.168.1.2 but since that address is assigned to a local interface on the master the packets will be discarded as or spoofed packets. The result is that "vm-002" (e.g. on that node) loses contact with the master;
I discovered this when testing PR I think I can restore the pre v1.23 behavior now, but I suffer from the problem described above.
I don't think your problem is precisely the same. To disable ipv6 or set nodePortAddresses can't really be used as work-arounds for this issue.

Thank you, You're right about IPv6 - disabling it didn't really help. I forgot to remove the I put in while testing with disabling IPv6. It was the that makes the difference. That said, it really appears to work if we explicitly put in the . Maybe I'm still missing something in the code, but if I read it correctly, here is where we take out the load balancer IP: It is called from the folllowing, but only if contains any zero CIDRs. If a node port address is explicitly provided and it is NOT a zero CIDR, then will not be called. We're planning to roll out this workaround before upgrading to a future Kubernetes release with your fix. We would much appreciate it if you could shed more light why it wouldn't work. We didn't want to change to iptables mode if this second workaround really works. Thank you so much!
You are right. To explicitly set in the kube-proxy config works as a work-around. Still the loadBalancerIP (which is also an address on a node) can't be used by the main k8s network, or the master must not be a node in the cluster as described in And... This will work up to and including K8s v1.25.x, but NOT v1.26.0! I haven't checked yet but I think it's because of
Since the loadBalancerIP that is owned by a node is assigned to on other nodes you must stop those other nodes from responding to ARP. For example with:
From my perspective, this is not a big , I have the whole architecture implemented in separate networks. I think it's worth adding this information somewhere in the documentation.
Thank you so much for the response, epesically the info about v1.26. We'll check it out for sure and will report back. Our node port use case is for single-node deployments only, so luckily there are no other nodes.


Question: "k8s.io/kubernetes/pkg/util/sets" "k8s.io/kubernetes/pkg/version" <del> "github.com/go-openapi/loads" "github.com/go-openapi/spec" "github.com/go-openapi/strfmt" "github.com/go-openapi/validate" </del> "github.com/stretchr/testify/assert" )

Haven't seen this one before and I am not sure it has any relation with the that hit it. https://k8s-
From what I can gather, it is being killed due to taking over 5 minutes which is roughly how long it takes (~300s) on successful unit builds.
bumping priority. I'm seeing this pretty frequently this morning. I see OpenAPI and associate you. You know something about this one?
Yes, I wrote this test. It was failing (timing out) for quite some time on jenkins but passing quite fast locally. I've disabled it but after a while it started passing and I assumed it was a configuration change on test machines. I will spend some time on it again to see if I can find the root cause and will disable it otherwise. This test is passing locally in about 15 second. Do you have any idea why it is timing out (or passing near to 300s) on jenkins machines? where do you see the successful test stats?
I see what you do. Locally it takes 25s, with bots it is 300s. My guess is that we are constrained by CPU and all the encode/decode is causing strain either on CPU or GC. The bots fail about 75% for me so I just keep retrying until success. Just had 2 this morning pass at 296s.
Curiously, it's taking about 20 seconds with Bazel. this might be worth looking at.
I think we run tests in Jenkins with , which might explain the slowness?
Nevermind, we hit it in a bazel run: https://k8s-
-race does kill the performance, 25s -166s which is on par with bazel.
is it possible to disable -race for this test? and should we do that if it is possible?
Now blocking the submit queue.
I think I am missing something here. I've checked some of the attached PRs. Links to the pkg/master folder in master branch has the change (new file called but non of the recent PRs I checked has the file in their branch: That means they didn't sync with master? What am I missing here?
OK. I see, the tests are patching the change of the PR into the master, so everybody should get that if they retest, but it look like my test is not blocking submit queue. there should be another issue.


Question: } } <del> tpExists, tpNeedsUpdate, err := gce.targetPoolNeedsUpdate(loadBalancerName, gce.region, affinityType) </del> <ins> tpExists, tpNeedsRecreation, err := gce.targetPoolNeedsRecreation(loadBalancerName, gce.region, apiService.Spec.SessionAffinity) </ins> if err != nil { return nil, err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area cloudprovider What happened: The cloud provider interface that should . The GCE provider does this for but not for . Every other in-tree provider implements by fully updating the load balancer including updated backend nodes: , , , And the following out-of-tree providers do as well and Oracle (not oss yet). What you expected to happen: I expect that all cloud providers implement by creating or fully updating the load balancer if it already exists. Related discussion: /cc
/assign Make sense to make implementation in GCE consistent with others for the ease of up-coming service controller enhancement. This sounds like a 1.10 task.
is this task blocking you on another attempt at refactoring the service controller sync logic?
yup


Question: framework.Failf("Error retrieving logs: %v", err) } <ins> if waitErr != nil { framework.Logf("Pod log:n%s", logOutput) framework.Failf("Pod did not stop running: %v", waitErr) } </ins> By("Verifying logs") verifyLogMessage(logOutput, "Accepted client connection") verifyLogMessage(logOutput, "Expected to read 3 bytes from client, but got 0 instead")

https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
So far I haven't found anything in the logs. The pfpod container starts, the e2e test receives all the data from it. At that time, the container should exit and the pod should be stopped. For some reason, either the container doesn't stop, or the kubelet doesn't see it stop. It's hard to tell which because the logs don't have enough information to make that determination.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
Might be the same issue as
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gce-etcd3 kubernetes-e2e-gci-gce Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
Adding to assignee list b/c of ssh tunnel flakes (i.e.

this issue has multiple, different root causes)
Builds: ci-kubernetes-e2e-gce-etcd3 ci-kubernetes-e2e-gci-gce-etcd3 ci-kubernetes-e2e-gci-gce-proto kubernetes-e2e-gce-etcd3 kubernetes-e2e-gci-gce-kubenet kubernetes-e2e-gci-gce-proto kubernetes-e2e-gci-gce-release-1.4 kubernetes-e2e-gci-gke-prod-smoke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
Closing this issue as the corresponding test has been removed from cc:


Question: p                 *ExecOptions name              string expectError       bool <del> expectedPod       string </del> <ins> expectedPod       *api.Pod </ins> expectedContainer string expectedArgs      []string }{

I feel like I should be able to do: but I get
or we have to do something like this: sounds reasonable to me. needs more comments.
All commands that accept resources should accept resource/NAME syntax. In this case for back compat we must only accept a single argument, is familiar with this.
Exactly. ResourceNames in the resource builder is what is needed here to handle the default resource (pod in this case) and provide the ability to specify more.
Related
If nobody is working on it, I'd like to pick the issue, pls.
I don't think anyone is. , Illia Khudoshyn wrote:
So could you please assign me?
I can't assign to just anyone, unfortunately, but by responding here, anyone else who might work on it will see that you are already in progress. , Illia Khudoshyn wrote:
please go ahead, you did own this now. :)
Has this been reverted again? I cannot use “pod/podname” for v1.11:
^ Seconding the above. I'm getting the same 'may not contain /' error. Naturally it's trivial to use unix commands to filter this, but this is such a basic feature and matches the way the rest of kubectl works, so it's quite a shocking omission.
as a unix novice who's also experiencing this problem, I'd love to know your workaround
If you use selectors, you can do something like this: Otherwise you'll need to manually grep, something like this: is the syntax in bash to run the command and use its output as part of another command. You can also use backticks to achieve the same effect but that's considered bad form by many (myself included). For example: If going the grep route, take care to make sure your regex will only return one result, otherwise you'll have to pipe it to xargs or something similar.
Just ran into this on a brand new default installation. Local Mac Docker for Desktop Enabled k8 in preferences error: invalid resource name "svc/broken-gorilla-postgresql": [may not contain '/']
FWIW you can use combined with the flag, e.g.
Had the same problem. Updating kubectl resolved the issue for me.


Question: } func BenchmarkPerfScheduling(b *testing.B) { <ins> dataItems := DataItems{Version: "v1"} </ins> tests := getSimpleTestCases(configFile) for _, test := range tests {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, &api.Pod{}, 0, <del> controllerFramework.ResourceEventHandlerFuncs{ </del> <ins> controllerframework.ResourceEventHandlerFuncs{ </ins> AddFunc: func(obj interface{}) { tracker.remember(obj.(*api.Pod), ADD) },

Forked from pointed out that all package names should be lower case, no under_scores or mixedCaps. The same rule applies to renaming imported packages. () generated a list of mixedCaps package names in our codebase: I will upload a PR later to rename kubeletTypes to kubetypes, and the rest is up for grabs.
sorry i have not saw this issue as i open a similar one(), do you mind i participate in this issue? i would happy to do some work for this. best regards.
of course, fixes are welcome. Thanks for picking up on this! We'll try to expedite the review and merge process of such PRs to avoid a lot of churns, but chances are that some rebases are needed :)
ok, i will send some fix sonner, and tag you for review. thanks again.
Not done:


Question: CreateLoadBalancer(*elb.CreateLoadBalancerInput) (*elb.CreateLoadBalancerOutput, error) DeleteLoadBalancer(*elb.DeleteLoadBalancerInput) (*elb.DeleteLoadBalancerOutput, error) DescribeLoadBalancers(*elb.DescribeLoadBalancersInput) (*elb.DescribeLoadBalancersOutput, error) <ins> AddTags(*elb.AddTagsInput) (*elb.AddTagsOutput, error) </ins> RegisterInstancesWithLoadBalancer(*elb.RegisterInstancesWithLoadBalancerInput) (*elb.RegisterInstancesWithLoadBalancerOutput, error) DeregisterInstancesFromLoadBalancer(*elb.DeregisterInstancesFromLoadBalancerInput) (*elb.DeregisterInstancesFromLoadBalancerOutput, error) CreateLoadBalancerPolicy(*elb.CreateLoadBalancerPolicyInput) (*elb.CreateLoadBalancerPolicyOutput, error)

BUG REPORT: /kind bug /sig aws What happened: Adding this annotation to an existing service: did not trigger an update on AWS ELB (no new tags). What you expected to happen: Expected to see those 2 new tags under the existing ELB: How to reproduce it (as minimally and precisely as possible): a service without annotation the annotation tags on AWS Anything else we need to know?: Creating a new service WITH the annotation will create the expected tags on AWS. Environment: Kubernetes version (use ): 1.7.9 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): Linux ip-xxxxx 4.4.78-k8s SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux Install tools: Others: Using Kops
+1
+1
+1
+1
+1
I've done some work on a fix for this, will try and get a PR submitted later today.


Question: } if len(instances) == 0 { glog.Warningf("the instance %s does not exist anymore", providerID) <del> return true, nil </del> <ins> // returns false, because otherwise node is not deleted from cluster // false means that it will continue to check InstanceExistsByProviderID return false, nil </ins> } if len(instances) > 1 { return false, fmt.Errorf("multiple instances found for instance: %s", instanceID)

/kind bug What happened: After upgrading 1.11.1 -1.12.0 , when a node gets terminated in AWS, the controller-manager doesn't delete the node in the API. This has lead to several side effect for us : pods don't get GC Prometheus alerts firing because 100 of nodes are in the NotReady state for a long period What you expected to happen: kube-controller-manager deletes the node when it stops reporting, after checking with the cloud provider. How to reproduce it (as minimally and precisely as possible): Looking at the code, I believe this is what introduced the behaviour change : - The node gets tainted with because it's in a Terminated state. But it doesn't get removed because it still "exists" in the cloud provider. The behaviour is probably correct for a shutdown node, but in AWS a terminated node can't be restarted so there's no good reason to keep it there ? Environment: Kubernetes version : 1.2.0 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS (sorry for the all the sigs, not sure who it belongs to) /sig aws /sig cloud-provider /sig cluster-lifecycle
as I see it should delete node if instance is in terminated state have you tested that change in that row? I do not see what could be the issue there. Or could it be that the is not in place for all binaries in 1.12.0?
I might be wrong, but I think that part of the code never gets executed once the node is tainted :
good finding. So we should remove terminated state from this row. Then it does not add shutdown taint if node is terminated - instead it will delete node. also this should return false.. because if it returns true node is not deleted
I had the same issue. After I scale down my worker node instance number from 3 to 1. One node is stuck in status and never got deleted. However, one interesting thing is one of the other node is removed successfully. do you see similar behavior too?
- can we make sure this gets cherry picked for 1.12 releases ? I think we already missed the cut for 1.12.2


Question: import ( "context" <ins> "errors" </ins> "fmt" "math/rand" "path/filepath"

master-informing: gce-cos-master-serial ! 2023-06-08 20:01:55 +0000 UTC { failed [FAILED] PV Create API error: Post "https://35.247.13.135/api/v1/persistentvolumes": context canceled In [DeferCleanup (Each)] at: test/e2e/storage/persistentvolumes- @ 06/09/23 18:05:53.165 } No response_ /sig storage cc
I think we did something wrong in /triage accepted
/assign
do you have a link to a full log? It's not clear where comes from (will look closer on Monday), but as the test is shutting down, not allowing new PVs to be created may be normal.
Never mind, the description has links to failing jobs.
ok - this looks like an actual bug with the code as written. what's happening is the "Stop and wait.." at line 492 is calling of the passed to the "goroutine [that] recycle[s] unbound PVs". Ordinarily that signal triggers the and the goroutine exits. In this case (and it will be flaky) it triggers while the case is running and it causes the on line 533 to error with . I don't know what the desired behavior is - should a error be ignored since we're signaling that it's time to leave? or should we actually wait for the to complete and use a different (timeout??) context for that instead.
one potential (if complex) fix is to pass two contexts down to the goroutine. one, to signal “you should stop when you are next at the select statement” and another , that is passed down to CreatePV that signals “stop immediately”. The would then take a Ginkgo context with a timeout. when the first launches it cancels and then if the timeout expires it cancels there might well be a simpler way to track all this but the very nature of what this goroutine is doing seems to need some sort of two-signal approach. alternatively: we can say that within this goroutine is always safe to ignore. it’s simpler as long as there are no test pollution concerns about forcibly canceling the CreatePV.
Patrick is the most knowledgable person in this area, we should wait for his response
Sounds good.

This might be a one off - or they may be a class of flaky tests that have a similar structure and so coming up with, and applying, a common pattern for signaling a background goroutine to exit gracefully could be useful.
I think ignoring the error is fine in this case. There is no reason for it to occur except when the test has already completed. -
sounds good - I suppose the only concern would be that forcibly aborting the might be an issue (e.g. the server's still working on things and we fail to clean up or race with it). but y'al lknow k8s far better than i do so maybe there's reasons to have confidence that it's a nonissue.
If the operation completes on the server side, then the other registered earlier will delete the PV.
last run was green let's see this is stable now, thanks for the help


Question: t.Fatalf("Failed to remove pods from desired state of world since they no longer exist") } <ins> // podWorker may call volume_manager WaitForUnmount() after we processed the pod in findAndRemoveDeletedPods() dswp.ReprocessPod(podName) dswp.findAndRemoveDeletedPods() // findAndRemoveDeletedPods() above must detect orphaned pod and delete it from the map if _, ok := dswp.pods.processedPods[podName]; ok { t.Fatalf("Failed to remove orphanded pods from internal map") } </ins> volumeExists := dswp.desiredStateOfWorld.VolumeExists(expectedVolumeName, "" /* SELinuxContext */) if volumeExists { t.Fatalf(

If a pod is created and then deleted an entry in processedPods map remains forever (sometimes, it's racy). The problem comes from the fact that calls volume_manager asynchronously. If the last call happens after populator cleaned up resources, an entry is to processedPods and will never be seen. An entry in processedPods map must be deleted if pod doesn't exist and pod runtime should removed according to . To verify that the issue really exists I applaied a simple printing resources in map. Here is a reproducer: brand-new fresh k8s cluster with entries in : a pod, check its uid: that this uid is accurately reported: the pod, wait for a few mins, check for again: The entry will be kept forever because kubelet doesn't know anything about this pod anymore: so no more events will ever come to populater for this uid, and the loop in won't find it because it was already deleted from . n/a <details</details<detailsany </details<details</details<details</details<details</details<details</details
/sig storage
/triage accepted
/priority impoerant-soon
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority impoerant-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
/assign
PR () fixed this issue. It is ready to merge since Mar 21.
/assign


Question: { description: "sandboxes for existing pods should not be garbage collected.", sandboxes: []sandboxTemplate{ <del> makeGCSandbox(pods[0], 0, runtimeapi.PodSandboxState_SANDBOX_READY, true), makeGCSandbox(pods[1], 0, runtimeapi.PodSandboxState_SANDBOX_NOTREADY, true), </del> <ins> makeGCSandbox(pods[0], 0, runtimeapi.PodSandboxState_SANDBOX_READY, true, 0), makeGCSandbox(pods[1], 0, runtimeapi.PodSandboxState_SANDBOX_NOTREADY, true, 0), </ins> }, containers: []containerTemplate{}, remain:     []int{0, 1},

kubernetes pre-submit unit tests for pass some times and fails some times causing the flakiness. Here is one such failure https://k8s- Passing: Failing: cc
Yep, I'm checking with this
Sorry for introducing a flakiness. Sent out the fix in .
/assign
Dup of . /close

https://k8s- Failed: TestSandboxGC {kuberuntime}
https://k8s- Failed: TestSandboxGC {kuberuntime}
/assign
Should've been fixed by


Question: return devicePath, nil } <del> err = b.mounter.FormatAndMount(devicePath, globalPDPath, b.fsType, nil) </del> <ins> err = b.mounter.FormatAndMount(devicePath, globalPDPath, b.fsType, b.mountOptions) </ins> if err != nil { return devicePath, fmt.Errorf("fc: failed to mount fc volume %s [%s] to %s, error %v", devicePath, b.fsType, globalPDPath, err) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently many in-tree storage plugins, including the iSCSI plugin, support mount options. The FC plugin does not. I have poked around the code enough to know where the plugin reports that mount options are not supported, but it isn't clear to me WHY they aren't supported. Why is this needed: Mount options can drastically change the performance and data persistence characteristics of a filesystem. For example, "noatime" can reduce metadata operations on frequently read data or "sync" can ensure all writes go immediately down to a backing storage array. I am working on a deployment across an FC backed cluster for which the the containerized application benefits from specific mount options, and I would guess other users might find similar use cases. It seems to me that if mount options are supportable on iSCSI, they should be equally supportable on FC, but I certainly recognize the possibility that I may be missing something. Otherwise, I'm guessing they simply haven't been a priority?
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue /help
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yeah, we just miss a volunteer that would implement this feature. It should be relatively easy and straightforward, following iscsi as an example.
I would like to work on it! As this my first issue I would like to get some help.
/assign sathish-thangaraj
GitHub didn't allow me to assign the following users: sathish-thangaraj. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.

For more information please see <detailsIn response to : /assign sathish-thangaraj Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
that's great! I was going to write detailed info how to fix this, however, I found out that mount options should work, at least in Kubernetes 1.13, see this commit: So there is a simple bug somewhere. Looking around the code, this code smells fishy: . Can you check with "BlockVolume" feature turned off in kubelet, i.e. ? Mount options should work there. And then it should be fairly easy to fix it in .
It definitely looks like there's an issue there. I don't think the kubelet ever made it that far in my initial testings, as mounting (with mountOptions listed in the PersistentVolume object) failed with the following message: I assume this is related to the following code: Is there a way to disable a check like this without rebuilding all of Kubernetes?
Good catch! No. IMO it should be enough to rebuild kubelet here and it's fast and easy: and find it somewhere in .
Thanks for the help so far I flipped the SupportsMountOption() bool to "true" and the BlockVolume feature gate to false. I was then able to mount an ext4 volume with the "noatime" and "nodiratime" option (confirmed on the worker node). However, I attempted to use the "nobarrier" option but the worker node mounted the volume without it. Finally, I attempted to use a completely made-up option ("hello"). The volume did not fail to mount (as is suggested in the Kubernetes documentation. Instead, it mounted without the "hello" options. I am attempting to locate useful logs to try to understand why some mount options are ignored or why incorrect mount options aren't causing failures, but the kubelet logs haven't been helpful thus far. Unfortunately I don't currently have any easy access to any iSCSI equipment for comparison.
anyone working on this?
is there anywhere that I could help on the issue?
First time contributor here - I just changed 2 lines of code to make FC similar to other volume types (eg. iSCSI).

I'm not sure about the mount option validation that mentions. Should that be a separate issue?
Apologies for the extreme delay in keeping up with this issue. I believe the issue with ignored mount options may be more generally related to the way the Linux mounter (/pkg/utils/mount/mount-) handles mount options, although I don't have enough experience with other volume plugins to speak authoritatively on the issue. I recommend we try to get the basic changes suggested above into the code, as my limited testing shows them to be effective for at least some mount options. I'd try to submit a pull request myself, but I don't want to take it from as they've already put forth the work to get it started.
To clarify my previous comment, during Attach, the FC volume plugin mounts an FC volume to a global path with no mount options. During Mount, the Linux mounter bind mounts (-o bind) and bind remounts (-o bind,remount) an FC volume to a Pod specific path with mount options. However, the bind remount action is only capable of applying a limited number of mount options (those not on the list are silently ignored). In order to expand our mount option capabilities, we would (I believe) need to use mount options in the Attach phase.
First time contributor, taking this one /assign
you are, of course, welcome to work on this issue. However, I wanted to let you know that I made code changes back in November that (I believe) address my previous comments. I only received approval from my employer in the last couple of days to make the contributions, or I would have submitted them already. I plan to clean things up and get a PR in sometime this afternoon. I am also a first time contributor, so we'll see how it goes. /assign
No problem at all dude. You take this one since you already made the changes. On Thu, 23 Jan 2020 at 18:47 Eric Weber <:


Question: NodeSyncPeriod:                    unversioned.Duration{Duration: 10 * time.Second}, ResourceQuotaSyncPeriod:           unversioned.Duration{Duration: 5 * time.Minute}, NamespaceSyncPeriod:               unversioned.Duration{Duration: 5 * time.Minute}, <del> PVClaimBinderSyncPeriod:           unversioned.Duration{Duration: 10 * time.Minute}, </del> <ins> PVClaimBinderSyncPeriod:           unversioned.Duration{Duration: 15 * time.Second}, </ins> HorizontalPodAutoscalerSyncPeriod: unversioned.Duration{Duration: 30 * time.Second}, DeploymentControllerSyncPeriod:    unversioned.Duration{Duration: 30 * time.Second}, MinResyncPeriod:                   unversioned.Duration{Duration: 12 * time.Hour},

Immediately after this test was reenabled in we got huge numbers of parallel flakes in . This might be a coincidence. I'll make a PR to move it to flaky. If things haven't improved after it's moved then I'll revert, else I'll assign this issue to !
Do you have any logs? references , but that has nothing to do with NFS nor PersistentVolumes.
. It's pretty clear this test caused the flakes we were seeing.
do you have any logs?
It hasn't flaked since your fix went in, but we should keep the issue open until it's moved out of the flaky suite.
The fix Joe Finney developed was merged on 5/30. The signature for the problem in the log file referenced here is: 6 18:56:07.895: Expected PersistentVolume to be bound, but got nil ClaimRef: kind:"" apiVersion:""` It looks like from this dashboard the last instance of this error log signature for the test PersistentVolumes NFS test to the flaky suite. was on 5/30 right around the time the code was merged. I found the off the dashboard at https://k8s- Since then the error has not occurred with that signature since I am tempted to close this bug, but keep the the test suite as marked "Flaky". The merge bot should start flagging the new bugs for this test going forward.
If it's no longer flaky we should move it out of the flaky suite. While it's in the flaky suite it won't block merges if it breaks and mergebot won't open bugs for it. I'm worried that if we just close this issue it'll sit in the flaky suite and break and noone will notice for a long time.
This bug has not been reproduced for past 2 months, can we close it?
It looks like the test is still in the flaky suite, although with a different name. Once it's moved out of there then this can be closed.
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention...
Not reproduced for three months, closing.

Consider reducing the PVClaimBinderSyncPeriod from 10 minutes down to something more reasonable, so that volumes don't sit around unbound for half an hour if it fails twice. This would reduce the impact of issues like until the binder/recycler/provosioner consolidation/refactor is complete. CC
+100
do you remember what was the reason for 10 minutes?
I checked provisioner and recycler, lowering the sync period should not harm them. Still, it will impact deleter a bit. Now we try to delete a volume three times every sync-period (10 minutes), mostly because if user deletes a pod that uses a volume and deletes also the PVC bound to it, the volume is still attached to the node for some time and all deletion attempts fail. I've seen AWS detach taking around 5 minutes in some cases. So, if we lower the timer, we should also increase the number of retries. Or have a different retry timer for deletion.


Question: // BetaAnnotationExternalTraffic is the beta version of AlphaAnnotationExternalTraffic. BetaAnnotationExternalTraffic = "service.beta.kubernetes.io/external-traffic" ) <del> // NeedsHealthCheck Check service for health check annotations func NeedsHealthCheck(service *api.Service) bool { // First check the alpha annotation and then the beta. This is so existing // Services continue to work till the user decides to transition to beta. // If they transition to beta, there's no way to go back to alpha without // rolling back the cluster. for _, annotation := range []string{AlphaAnnotationExternalTraffic, BetaAnnotationExternalTraffic} { if l, ok := service.Annotations[annotation]; ok { if l == AnnotationValueExternalTrafficLocal { return true } else if l == AnnotationValueExternalTrafficGlobal { return false } else { glog.Errorf("Invalid value for annotation %v: %v", annotation, l) } } } return false } // GetServiceHealthCheckNodePort Return health check node port annotation for service, if one exists func GetServiceHealthCheckNodePort(service *api.Service) int32 { if !NeedsHealthCheck(service) { return 0 } // First check the alpha annotation and then the beta. This is so existing // Services continue to work till the user decides to transition to beta. // If they transition to beta, there's no way to go back to alpha without // rolling back the cluster. for _, annotation := range []string{AlphaAnnotationHealthCheckNodePort, BetaAnnotationHealthCheckNodePort} { if l, ok := service.Annotations[annotation]; ok { p, err := strconv.Atoi(l) if err != nil { glog.Errorf("Failed to parse annotation %v: %v", annotation, err) continue } return int32(p) } } return 0 } </del>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS m4.2xlarge OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-36-generic Install tools: kubeadm Others: 1 master and 1 minion What happened: This states that if using kube-proxy will only proxy requests to local endpoints, never forwarding traffic to other nodes and thereby preserving the original source IP address. However when I annotate my NodePort type service with this annotation I see the following error in the logs of kube-proxy of the node where the only pod of the service is running: After this message the service stops working outside of the cluster on the specified NodePort. What you expected to happen: I expected to see that only the with the pod on it serves traffic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
Sorry, I have found the issue, I had a setup where the none of the node names were the same as the hostname of the kube-proxy pod and this caused the issue, where none of the Pods running on a hosts were recognised as local pods. I have fixed this issue by manually setting the hostname with in kube-proxy. However I still insist that this error message is misleading and shouldn't be printed in case of NodePort type services, but only for just LoadBalancers.
I met the same problem in the aws cloud providers, by adding - the hostname-override to solve this problem. good


Question: // Raw is the underlying serialization of this object. // // TODO: Determine how to detect ContentType and ContentEncoding of 'Raw' data. <del> Raw []byte `protobuf:"bytes,1,opt,name=raw"` </del> <ins> Raw []byte `json:"-" protobuf:"bytes,1,opt,name=raw"` </ins> // Object can hold a representation of this extension - useful for working with versioned // structs. Object Object `json:"-"`

The Service Catalog includes the type in . The openapi schema generated for it is wrong. It has two problems: it includes the field, which is marked as required it doesn't allow additional properties to be included under it Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT /kind bug What happened: If the Service Catalog openapi schema is exposed to kubectl (this currently requires a few tweaks and a patch), it reports validation errors when creating a ServiceInstance resource. What you expected to happen: The schema should allow to include arbitrary properties and not require the property to be included. How to reproduce it (as minimally and precisely as possible): not easy at the moment (I'll add instructions later if I'm unable to fix this myself) Anything else we need to know?: I'd like to tackle this problem myself, but I need guidance. Should openapi-gen handle this automatically for all occurences of RawExtension, or should each field of type be json-tagged individually? Environment: Kubernetes version (use ):
I've heard you're the goto guy for the openapi-gen. Please help? :)
/sig api-machinery
OK, I've found I can specify a custom schema for RawExtension the way it's done in Time and MicroTime. The remaining problem is how to specify that the RawExtension object can be of any type. Using is not supported currently - if I use it in the schema, the API server dies with:
/assign /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
We've just hit this. /remove-lifecycle rotten /lifecycle frozen
hitting this
According to this: the raw extension can define its openapi type as: or alternatively what do you think?

do you still work on this or you want me to send a PR?
Would changing the definition of RawExtension to this help the situation? (adding ) I noticed that this removes Raw from the openapi defintion of it, without changing how it is marshalled or unmarshalled (since we have custom functions for that) I'll try making a PR for this