Question: # If the URL is for GCS and the node has dev storage scope, add the # service account OAuth2 bearer token to the request headers. # https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#applications <del> if (($Url -match "^https://storage`.googleapis`.com.*") -And $(Check-StorageScope)) { </del> <ins> if (($Url -match "^https://storage`.googleapis`.com.*") -and $(Check-StorageScope)) { </ins> $httpClient.DefaultRequestHeaders.Add("Authorization", "Bearer $(Get-Credentials)") } # Attempt to download the file $httpResponseMessage = $httpClient.GetAsync([System.Uri]::new($Url)) $httpResponseMessage.Wait() <del> if (-Not $httpResponseMessage.IsCanceled) { </del> <ins> if (-not $httpResponseMessage.IsCanceled) { </ins> # Check if the request was successful. # # DO NOT replace with EnsureSuccessStatusCode(), it prints the # OAuth2 bearer token. <del> if (-Not $httpResponseMessage.Result.IsSuccessStatusCode) { </del> <ins> if (-not $httpResponseMessage.Result.IsSuccessStatusCode) { </ins> $statusCode = $httpResponseMessage.Result.StatusCode throw "Downloading ${Url} returned status code ${statusCode}, retrying." }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: GCE Windows cluster bring up fails on some versions of Windows. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: This is a regression caused by Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign jeremyje /sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign pjh
/unassign pjh


Question: "kubeproxy_iptables_localhost_nodeports_accepted_packets_total", "Number of packets accepted on nodeports of loopback interface", nil, nil, metrics.ALPHA, "") <del> LocalhostNodePortAcceptedNFAcctCounter     = "localhost_nps_accepted_pkts" localhostNodePortsAcceptedMetricsCollector = newNFAcctMetricCollector(LocalhostNodePortAcceptedNFAcctCounter, localhostNodePortsAcceptedPacketsDescription) </del> <ins> LocalhostNodePortAcceptedNFAcctCounter = "localhost_nps_accepted_pkts" </ins> ) var registerMetricsOnce sync.Once

The module now logs two errors during module initialization. Simply loading the module will cause two errors to be printed to stderr if the code is not run as root, or is run on a node without nfacct support. This is due to being used when creating Prometheus metrics during module variable initialization, which calls if the nfacct client cannot be initialized. This regression was introduced by: K3s ships all the various Kubernetes components bundled in a single multicall binary, which means that our now logs these errors when it is run as a non-root user. No error logs on module init cc <details</detailsn/a n/a n/a n/a n/a area/proxy sig/network
/sig network
/triage accepted /assign
I was about to fix this, this initialization happens in the unit tests as well .
thanks for the quick response! Do you think you'll have a patch that we could use temporarily for k3s until v1.31.1 is out? We were considering just removing the call for our initial 1.31 release, unless you have a better suggestion.
can you confirm is fine?


Question: PREPULL_E2E_IMAGES="${PREPULL_E2E_IMAGES:-true}" # Evict pods whenever compute resource availability on the nodes gets below a threshold. <del> EVICTION_HARD="${EVICTION_HARD:-memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%}" </del> <ins> EVICTION_HARD="${EVICTION_HARD:-memory.available<250Mi,nodefs.available<10%,nodefs.inodesFree<5%}" </ins> # Optional: custom scheduling algorithm SCHEDULING_ALGORITHM_PROVIDER="${SCHEDULING_ALGORITHM_PROVIDER:-}"

Clusters on GCE set eviction thresholds for disk and inodes. Given that reclaiming disk and inodes is slow, we should set minimum reclaim thresholds too. Essentially, set . I assume percentages are supported for min reclaims too. Do we have system tests for min reclaim? cc
I just took a glance at the parser for eviction minimum reclaim, it doesn't look like it tries to parse percentages. can you confirm?
did percentage support, and we did not extend it to min reclaim. It's a good idea to do though. On Tuesday, September 13, 2016, Michael Taufen wrote:
Hmm.. In that case for v1.4, can we set the min reclaim to ? What values are you planning on using in Open Shift ?
This should fix this issue to avoid node condition being flip-flop, but I think it shouldn't block 1.4 release.
More than flip-flop, it will help the node recover from disk pressure condition sooner. , Dawn Chen wrote:
/cc
Spawned to add percentage support for min reclaim. I will follow up with our proposed default for nodefs later
This needs to be triaged as a release-blocker or not for 1.5
can you check if we have set default values for disk and inode based evictions in kubelet? If not, we should add them.
I believe our current default value for the --eviction-hard flag is "memory.available<100Mi", found in Our tests currently run using: "memory.available<250Mi,nodefs.available<10%,nodefs.inodesFree<5%", found in What should I change the default to? This issue is about minimum reclaim, should I look into changing those flags as well?
It doesn't appear that we have defaults for minimum reclaim, should I add minimum reclaim for memory in addition to disk and inodes?
I think GCE defaults do not apply globally, we should change what GCE does, and handle setting global defaults in 1.6.
the global defaults are expected to be based on our OSS tests. Our original plan was to roll out disk eviction by default in v1.5. This is not specific to GCE.
Can you make a call on if this is a blocker or non-blocker for 1.5 by EOD Friday please


Question: // TestRegions does basic checking of region verification / addition func TestRegions(t *testing.T) { <del> RecognizeWellKnownRegions() </del> <ins> recognizeWellKnownRegions() </ins> tests := []struct { Add            string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: From /kind cleanup What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cloud-provider /assign


Question: func TestAnonymous(t *testing.T) { var a authenticator.Request = NewAuthenticator() <del> r, ok, err := a.AuthenticateRequest(nil) </del> <ins> r, ok, err := a.AuthenticateRequest(&http.Request{}) </ins> if err != nil { t.Fatalf("Unexpected error %v", err) }

Semantics are going to vary. We need to cross off these authenticators: [x] if ctx has an audience limitation, verify the token audience and echo in the response [x] take implicit audiences at WebhookTokenAuthenticator construction time if ctx is audience-limited, add into the tokenreview object if the tokenreview returns with an audience set that matches the ctx, copy into the response and return success if the tokenreview returns without an audience set, ensure the ctx audiences intersect with the implicit audiences taken at WebhookTokenAuthenticator construction time, and pick one to add into the response [x] echo audience in response if present in ctx [x] echo audience in response if present in ctx [x] check that the the order of (audience in the context) ∩ ( (implicit api audiences) ∪ (clientids) ) 0 Take implicit audiences at Authenticator construction time, and if ctx is audience-limited, ensure it intersects with the implicit audiences, and echo an audience in the response. [x] [x] [x] [x] [x] [x] Do nothing? [x] [x] [x] [x] [x] [x] [x] note: we already keyed the cache with the audience. [x]
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: gcloud compute firewall-rules delete   --project "${PROJECT}"  --quiet  <del> "${MASTER_NAME}-https" || true </del> <ins> "${MASTER_NAME}-https" </ins> # Delete firewall rule for minions. gcloud compute firewall-rules delete   --project "${PROJECT}"  --quiet  <del> "${MINION_TAG}-all" || true </del> <ins> "${MINION_TAG}-all" </ins> # Delete routes. local -a routes

'kubernetes-soak-weekly-deploy' sometimes fails to deploy because the clean-up cluster step fails to delete the template: Then the subsequent cluster step fails to create the template, because it already exists: It keeps retrying and failing, because the template already exists, until it times out and the deployment fails. Suggested fixes: Cluster down should try harder to delete the template (maybe wait/retry). Cluster up should recognize that the template exists, and try to delete/recreate, or just re-use the existing.
Looks like ran in to this on Monday, May 11, 2015 around 1:00 PM PST. I'll assign this to my self and take a crack at fixing it.
This is causing most of our e2e tests to fail in continuous integration as well as users private e2e test clusters. Bumping to priority/p0 and v1.0 milestone. Saad thinks he's figured it out, and is working on a fix as we speak.
In essence... Saad: "I think what's happening is the previous clean up step to delete the managed instance group happens asynchronously, therefore there is a race condition to delete the template. Working on a fix".


Question: DNSDomain      string } <ins> // NewWindowsModel returns a model specific to windows testing. func NewWindowsModel(namespaces []string, podNames []string, ports []int32, dnsDomain string) *Model { return NewModel(namespaces, podNames, ports, []v1.Protocol{v1.ProtocolTCP}, dnsDomain) } // GetProbeTimeoutSeconds returns a timeout for how long the probe should work before failing a check, and takes windows heuristics into account, where requests can take longer sometimes. func (m *Model) GetProbeTimeoutSeconds() int { timeoutSeconds := 1 if framework.NodeOSDistroIs("windows") { timeoutSeconds = 3 } return timeoutSeconds } // GetWorkers returns the number of workers suggested to run when testing, taking windows heuristics into account, where parallel probing is flakier. func (m *Model) GetWorkers() int { numberOfWorkers := 3 if framework.NodeOSDistroIs("windows") { numberOfWorkers = 1 // See https://github.com/kubernetes/kubernetes/pull/97690 } return numberOfWorkers } </ins> // NewModel instantiates a model based on: // - namespaces // - pods

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Ive done preliminary testing with on calico powered clusters on some windows variants and found that the probing seems to work. Can we add wrapper to netpol suite so that we can run it on windows clusters ? Why is this needed: We need this because otherwise, CNIs such as EKS will give us This error is caused when, for example, pods that are landing on a windows node havent defined their target platform. Somehow i guess pods which are landing on linux nodes dont have this issue. Its also needed for the obvious reason that we probably want separate linux and windows test suites for policies, in general, even though we expect the logic to be identical (modulo hostports , which we dont currently test)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/assign
grepping around, I can't figure out where those are supposed to go. Can you give me pointers?
(resolved in slack, matts digging into this)
ok, these are running now on eks, lots of failures. will dig
/assign
the netpol are not being applied for the ns: Instead of failing because should not be able to access , the connection succeeds.


Question: } return obj, nil } creating = false <ins> newVersion, err := e.Helper.ResourceVersioner.ResourceVersion(obj) if err != nil { return nil, err } if newVersion != version { // TODO: return the most recent version to a client? return nil, kubeerr.NewConflict(e.EndpointName, name, fmt.Errorf("the resource was updated to %d", version)) } </ins> if err := rest.BeforeUpdate(e.UpdateStrategy, ctx, obj, existing); err != nil { return nil, err } <del> // TODO: expose TTL </del> return obj, nil }) if err != nil { if creating { err = etcderr.InterpretCreateError(err, e.EndpointName, name)

The use of AtomicUpdate by genericetcd doesn't handle conflicting writes by rejecting that to a user. It should. Eventually, we may want to hide write conflicts to /pods when status has changed, and to spec when /pods/status changes, but that requires us to have two version counters and that's not worth the effort. Fix incoming.
Also we need a unit/integration test for this.


Question: ) var onlyOneSignalHandler = make(chan struct{}) <ins> var shutdownHandler chan os.Signal </ins> // SetupSignalHandler registered for SIGTERM and SIGINT. A stop channel is returned // which is closed on one of these signals. If a second signal is caught, the program

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I configured Kubelet to run as a service on Windows, with a setting to restart on failure (). I noticed that it restarts even when I explicitly stop the service with , even though the documentation states the following (I left it at , the default): What you expected to happen: I expected the Kubelet to only restart when killed unexpectedly, per above docs. Anything else we need to know?: I think the problem might be due to this part of our service control integration, but I'm not really sure: It looks like we send and then , but the official golang example sends and just returns without exiting. (). Is this difference meaningful? Kubernetes version (use ): 1.13.2 Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Windows /cc /sig node /sig windows
The same should apply to kube-proxy as well.
i believe you implemented this functionality. can you please take a look? cc:


Question: <del> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package service import ( "context" "encoding/json" "fmt" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/strategicpatch" v1core "k8s.io/client-go/kubernetes/typed/core/v1" ) // patch patches service's Status or ObjectMeta given the origin and // updated ones. Change to spec will be ignored. func patch(c v1core.CoreV1Interface, oldSvc *v1.Service, newSvc *v1.Service) (*v1.Service, error) { // Reset spec to make sure only patch for Status or ObjectMeta. newSvc.Spec = oldSvc.Spec patchBytes, err := getPatchBytes(oldSvc, newSvc) if err != nil { return nil, err } return c.Services(oldSvc.Namespace).Patch(context.TODO(), oldSvc.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status") } func getPatchBytes(oldSvc *v1.Service, newSvc *v1.Service) ([]byte, error) { oldData, err := json.Marshal(oldSvc) if err != nil { return nil, fmt.Errorf("failed to Marshal oldData for svc %s/%s: %v", oldSvc.Namespace, oldSvc.Name, err) } newData, err := json.Marshal(newSvc) if err != nil { return nil, fmt.Errorf("failed to Marshal newData for svc %s/%s: %v", newSvc.Namespace, newSvc.Name, err) } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, v1.Service{}) if err != nil { return nil, fmt.Errorf("failed to CreateTwoWayMergePatch for svc %s/%s: %v", oldSvc.Namespace, oldSvc.Name, err) } return patchBytes, nil } </del>

PR service patch methods to Once there is a release with that PR included, the code in service-controller and legacy-cloud-providers/gce can be updated to use those. /area platform/gce /assign
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Created a PR to use these new functions in Service-controllers.


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "fmt" "time" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" "k8s.io/api/core/v1" storage "k8s.io/api/storage/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" ) const ( resizePollInterval = 2 * time.Second // total time to wait for cloudprovider or file system resize to finish totalResizeWaitPeriod = 20 * time.Minute ) var _ = SIGDescribe("Volume expand [Feature:ExpandPersistentVolumes] [Slow]", func() { var ( c           clientset.Interface ns          string err         error pvc         *v1.PersistentVolumeClaim resizableSc *storage.StorageClass ) f := framework.NewDefaultFramework("volume-expand") BeforeEach(func() { framework.SkipUnlessProviderIs("aws", "gce") c = f.ClientSet ns = f.Namespace.Name framework.ExpectNoError(framework.WaitForAllNodesSchedulable(c, framework.TestContext.NodeSchedulableTimeout)) test := storageClassTest{ name:      "default", claimSize: "2Gi", } resizableSc, err = createResizableStorageClass(test, ns, "resizing", c) Expect(err).NotTo(HaveOccurred(), "Error creating resizable storage class") Expect(*resizableSc.AllowVolumeExpansion).To(BeTrue()) pvc = newClaim(test, ns, "default") pvc.Spec.StorageClassName = &resizableSc.Name pvc, err = c.CoreV1().PersistentVolumeClaims(pvc.Namespace).Create(pvc) Expect(err).NotTo(HaveOccurred(), "Error creating pvc") }) AfterEach(func() { framework.ExpectNoError(framework.DeletePersistentVolumeClaim(c, pvc.Name, pvc.Namespace)) framework.ExpectNoError(c.StorageV1().StorageClasses().Delete(resizableSc.Name, nil)) }) It("Verify if editing PVC allows resize", func() { By("Waiting for pvc to be in bound phase") pvcClaims := []*v1.PersistentVolumeClaim{pvc} pvs, err := framework.WaitForPVClaimBoundPhase(c, pvcClaims, framework.ClaimProvisionTimeout) Expect(err).NotTo(HaveOccurred(), "Failed waiting for PVC to be bound %v", err) Expect(len(pvs)).To(Equal(1)) By("Creating a pod with dynamically provisioned volume") pod, err := framework.CreatePod(c, ns, nil, pvcClaims, false, "") Expect(err).NotTo(HaveOccurred(), "While creating pods for resizing") defer func() { err = framework.DeletePodWithWait(f, c, pod) Expect(err).NotTo(HaveOccurred(), "while cleaning up pod already deleted in resize test") }() By("Expanding current pvc") newSize := resource.MustParse("6Gi") pvc, err = expandPVCSize(pvc, newSize, c) Expect(err).NotTo(HaveOccurred(), "While updating pvc for more size") Expect(pvc).NotTo(BeNil()) pvcSize := pvc.Spec.Resources.Requests[v1.ResourceStorage] if pvcSize.Cmp(newSize) != 0 { framework.Failf("error updating pvc size %q", pvc.Name) } By("Waiting for cloudprovider resize to finish") err = waitForControllerVolumeResize(pvc, c) Expect(err).NotTo(HaveOccurred(), "While waiting for pvc resize to finish") By("Checking for conditions on pvc") pvc, err = c.CoreV1().PersistentVolumeClaims(ns).Get(pvc.Name, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred(), "While fetching pvc after controller resize") inProgressConditions := pvc.Status.Conditions Expect(len(inProgressConditions)).To(Equal(1), "pvc must have resize condition") Expect(inProgressConditions[0].Type).To(Equal(v1.PersistentVolumeClaimResizing), "pvc must have resizing condition") By("Deleting the previously created pod") err = framework.DeletePodWithWait(f, c, pod) Expect(err).NotTo(HaveOccurred(), "while deleting pod for resizing") By("Creating a new pod with same volume") pod2, err := framework.CreatePod(c, ns, nil, pvcClaims, false, "") Expect(err).NotTo(HaveOccurred(), "while recreating pod for resizing") defer func() { err = framework.DeletePodWithWait(f, c, pod2) Expect(err).NotTo(HaveOccurred(), "while cleaning up pod before exiting resizing test") }() By("Waiting for file system resize to finish") pvc, err = waitForFSResize(pvc, c) Expect(err).NotTo(HaveOccurred(), "while waiting for fs resize to finish") pvcConditions := pvc.Status.Conditions Expect(len(pvcConditions)).To(Equal(0), "pvc should not have conditions") }) }) func createResizableStorageClass(t storageClassTest, ns string, suffix string, c clientset.Interface) (*storage.StorageClass, error) { stKlass := newStorageClass(t, ns, suffix) allowExpansion := true stKlass.AllowVolumeExpansion = &allowExpansion var err error stKlass, err = c.StorageV1().StorageClasses().Create(stKlass) return stKlass, err } func expandPVCSize(origPVC *v1.PersistentVolumeClaim, size resource.Quantity, c clientset.Interface) (*v1.PersistentVolumeClaim, error) { pvcName := origPVC.Name updatedPVC := origPVC.DeepCopy() waitErr := wait.PollImmediate(resizePollInterval, 30*time.Second, func() (bool, error) { var err error updatedPVC, err = c.CoreV1().PersistentVolumeClaims(origPVC.Namespace).Get(pvcName, metav1.GetOptions{}) if err != nil { return false, fmt.Errorf("error fetching pvc %q for resizing with %v", pvcName, err) } updatedPVC.Spec.Resources.Requests[v1.ResourceStorage] = size updatedPVC, err = c.CoreV1().PersistentVolumeClaims(origPVC.Namespace).Update(updatedPVC) if err == nil { return true, nil } framework.Logf("Error updating pvc %s with %v", pvcName, err) return false, nil }) return updatedPVC, waitErr } func waitForControllerVolumeResize(pvc *v1.PersistentVolumeClaim, c clientset.Interface) error { pvName := pvc.Spec.VolumeName return wait.PollImmediate(resizePollInterval, totalResizeWaitPeriod, func() (bool, error) { pvcSize := pvc.Spec.Resources.Requests[v1.ResourceStorage] pv, err := c.CoreV1().PersistentVolumes().Get(pvName, metav1.GetOptions{}) if err != nil { return false, fmt.Errorf("error fetching pv %q for resizing %v", pvName, err) } pvSize := pv.Spec.Capacity[v1.ResourceStorage] // If pv size is greater or equal to requested size that means controller resize is finished. if pvSize.Cmp(pvcSize) >= 0 { return true, nil } return false, nil }) } func waitForFSResize(pvc *v1.PersistentVolumeClaim, c clientset.Interface) (*v1.PersistentVolumeClaim, error) { var updatedPVC *v1.PersistentVolumeClaim waitErr := wait.PollImmediate(resizePollInterval, totalResizeWaitPeriod, func() (bool, error) { var err error updatedPVC, err = c.CoreV1().PersistentVolumeClaims(pvc.Namespace).Get(pvc.Name, metav1.GetOptions{}) if err != nil { return false, fmt.Errorf("error fetching pvc %q for checking for resize status : %v", pvc.Name, err) } pvcSize := updatedPVC.Spec.Resources.Requests[v1.ResourceStorage] pvcStatusSize := updatedPVC.Status.Capacity[v1.ResourceStorage] //If pvc's status field size is greater than or equal to pvc's size then done if pvcStatusSize.Cmp(pvcSize) >= 0 { return true, nil } return false, nil }) return updatedPVC, waitErr } </ins>

We should have e2e feature for resizing in GCE environment. Also add unit tests for node restrictions.
/sig storage /assign


Question: glog.V(4).Infof("Copying %s: %s -> %s", direction, src.RemoteAddr(), dest.RemoteAddr()) n, err := io.Copy(dest, src) if err != nil { <del> glog.Errorf("I/O error: %v", err) </del> <ins> if !isClosedError(err) { glog.Errorf("I/O error: %v", err) } </ins> } glog.V(4).Infof("Copied %d bytes %s: %s -> %s", n, direction, src.RemoteAddr(), dest.RemoteAddr()) <del> dest.CloseWrite() src.CloseRead() </del> <ins> dest.Close() src.Close() </ins> } // udpProxySocket implements proxySocket.  Close() is implemented by net.UDPConn.  When Close() is called,

While investigating a socket leak in our kubernetes-deployed application, I have found out that the kube-proxy also seems to have a serious leak of its own. In two of my three nodes, out of the ~44K connections of the kube-proxy .... ... about 65% are legit established connections caused by the aforementioned leak ... ... but ~14K are in FINWAIT2 state ... ... and ~1K are in CLOSEWAIT. Here's some debug info: Gist with service descriptions, node descriptions and iptables rules from node (we are running kubernetes 1.1 in AWS): Goroutine dump of kube-proxy at node , obtained with : at node : has a value of 60 in all my nodes. After deploying the fix for application-level leak and waiting 10 minutes, according to the number of ESTABLISHED connections is ~100 in all nodes as expected but, in the two affected nodes: The CLOSEWAIT connections only halve from 1K to 500 The FINWAIT2 ones also halve from 14K to 7K I obtained new goroutine and netstat dumps, in case it helps: Also, it may be valuable to know that the application-level leaks were caused by a websocket proxy (part of the pods) which failed to clean up connections when the client-side closed the underlying TCP socket. Both the app-mapper and the destination are accessed through kubernetes services, which means that the kube-proxy is caught up in-between: The two kube-proxy instances affected live in the same nodes where the two app-mapper pods live: and
Uhm, I have noticed that netstat somehow wasn't reporting all the sockets. Whilst lsof does: The extra entries seem to correspond to : So, I think my analysis above still applies. Here's the lsof listing in case it helps:
Can you post anything interesting from kube- Does anything appear to be spinning? Maybe try bumping the kube-proxy --v flag to 4 and reproducing , Alfonso Acosta wrote:
For record: The majority of hung goroutines (about 2/3 of them) are at go copyBytes("from backend", in, out, &wg)go copyBytes("to backend", out, in, &wg), both of which call down to . I note that we half-close (CloseWrite(), CloseRead()) connections in each direction.

I'm not sure I recall why exactly. I could try some tweaks around here to get more info - are you willing to run a test binary for me? , Tim Hockin wrote:
The kube-proxy logs were empty (rotated by I guess) but I can try to reproduce in a separate cluster. It would be good to know, 7K are a lot plus they escape netstat (and I would bet that the same happens with most monitoring systems). Sure
I have reproduced it with our unfixed, leaky app and kube-proxy running with (in fact I can reproduce it systematically). I can confirm that, for each descriptor pair our application leaks, kube-proxy gets one socket in CLOSEWAIT and another one in FINWAIT2. Again, to give more context on our application: What's happening is that the frontend closes the client socket, but the app-mapper (which is proxying the frontend connection to the destination) doesn't close its connection pair. The CLOSEWAIT socket in kube-proxy corresponds to the frontend ("from" side) socket and the FINWAIT2 to the app-mapper ("to" side). My theory is that, when cleaning up a connection pair (due to the client closing its connection), kube-proxy expects the other end in the "to" connection to close it right away, but when it doesn't (due to the leak in this case) it stays in the FINWAIT2 state forever and leaving the "from" side in CLOSEWAIT. I think that, unlike what stated in , kube-proxy should probably timeout the connections on the "to" end after certain time. I don't see a point to wait for the "to" end if the "from" end was closed. I am not familiar with kube-proxy's internals so my reasoning may be an oversimplification or simply wrong. Here is the netstat listing from one of the affected nodes: Here are the logs of the kube-proxy running in the same node: Here is the kube-proxy go-routine dump: Here is the node description: Here are the service descriptions:
After having a look at the code, causing a couple of leaks and removing all pods to reduce the connections of the kube-proxy to a minimum, I have a pretty good idea of what's happening. The exits, but its doesn't.

I need to confirm this, but my guess is that the in the copier are not enough to guarantee that in the routine returns if the remote end doesn't close the socket. You may find the full go-routing stack here:
Good news - I have a clue what is happening (I think). I am pretty sure it is the half-close logic we have - it's technically possible to close just one direction of a TCP session. If the client closes the socket, the proxy gets a signal that the client read-side closed and closes the backend's write-side. But it does not get a signal that the client's write side closed (because POSIX, that's why). If the backend responds by closing the socket (since it knows if it has more data to send or not), the proxy gets a signal that the backend read-side closed and closes the client's write side. If the backend DOES NOT close the socket, the proxy waits forever. It does not get a signal that the client's write side closed. So the easy fix is to not try the half-close stuff and just say that closed means closed, lingering data be damned. The more robust fix might be to start a timeout on the backend once the client disconnects. Here is a kube proxy binary which just closes the connection - try it out? , Alfonso Acosta wrote:
Specifically run with --v=4 and let me see logs :) , Tim Hockin wrote:
Yep, that matches my observations above. I haven't run your binary but in parallel to your investigations I have come up with a simple repro which does exactly that. For every leak created by the service I get a / pair in the kube-proxy: I can run your binary tomorrow if you still need me to, but I think this confirms the theory. I believe that the opposite timer (on the client once the backend disconnects) would also be necessary
I found some time and got the logs from a recent unmodified kube-proxy binary and the one provided while causing a leak in the leaker service: They confirm that the problem comes from the current use of and .
Thanks for the confirmation. I can't say for sure whether these are the exact same problem, but they are suspiciously similar. I'll try to work up a real patch tomorrow. , Alfonso Acosta wrote:
Patch is out for review. Thanks for the help!


Question: if mapping.Kind != "ReplicationController" { usageError(cmd, "%s does not specify a valid ReplicationController", filename) } <ins> if oldName == newName { usageError(cmd, "%s cannot have the same name as the existing ReplicationController %s", filename, oldName) } </ins> err = CompareNamespaceFromFile(cmd, namespace) checkErr(err)

I believe the problem is that requires a schema of but that Kubernetes does not have the ability to handle that schema yet. I just want to confirm this is true however. So with a schema like: It fails when used like so: If I try a schema that looks like: then the error is: The error generation comes from: Am I correct that it is impossible to use the command against ?
For clarity's sake here is my version info:
Rename cloud-delivery-test to cloud-delivery-test2 or something else. kubectl rollingupdate doesn't update an existing ReplicationController, it replaces it with a new one.
Doc:
Looking into this now. I am able to run kubectl rollingupdate against a 0.8.0 cluster using a v1beta1 schema. So it may just be broken for v1beta2/v1beta3.
Also, v1beta3 isn't enabled by default. Here's a message about how to enable it: !topic/kubernetes-dev/ZqbLnrg5--8 It's user error. I think we need a clearer error message when the 2 replication controllers have the same name.
good catch. can you confirm the command works with a distinctly named controller?
I see guys, ok, thank-you. Until patch I don't think this was made clear anywhere : ) I will try this out tomorrow and confirm but surely it will work. Thanks again.
Thanks It works differently from kubecfg rollingupdate, so I'm sure other people will encounter the same issue.


Question: "k8s.io/apiserver/pkg/admission" genericadmissioninitializer "k8s.io/apiserver/pkg/admission/initializer" resourcequotaapi "k8s.io/apiserver/pkg/admission/plugin/resourcequota/apis/resourcequota" <ins> v1 "k8s.io/apiserver/pkg/admission/plugin/resourcequota/apis/resourcequota/v1" </ins> "k8s.io/apiserver/pkg/admission/plugin/resourcequota/apis/resourcequota/validation" quota "k8s.io/apiserver/pkg/quota/v1" "k8s.io/apiserver/pkg/quota/v1/generic"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It appears that validating admission requests for server-side apply are populating the for requests to create Namespace objects. Non server-side apply requests don't do this, and a missing namespace field is the only way to distinguish cluster-scoped resources from namespace-scoped resources. This appears to be a breaking behavioral change. See What you expected to happen: Requests to create namespaces would have a consistent signature between server-side apply and non-server-side apply contexts. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: kind OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig-api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/wg api-expression
/assign
/assign
Requests made to namespaced URLs (URLs which contain a segment) populate the attribute in AdmissionReview requests. The documentation of the AdmissionReview field is as follows: From Update/Patch/Delete requests to specific namespaces have consistently populate this attribute for modifications to Namespace objects. Server-side apply is a type of patch request, so even when it is creating an object, that attribute will be populated. The difference is caused by a difference in the URL used by POST and apply-based creates. Creation via POST requests are made against the endpoint, so no attribute was populated. Namespace computed from request path: Namespace admission attributes for POST-based creates: Creation via server-side apply is done via a PATCH request to , so the attribute is populated. Namespace computed from request path: Namespace admission for apply-based creates: This is currently working as intended. If we want to make the behavior consistent between the two, I'd consider populating the namespace attribute for POST-based creation of Namespace objects.

Thanks for the thorough analysis Jordan, the problem OP is trying to solve is that they want to know if the object is namespaced or not, and used the presence of the field to detect that. According to your analysis, that seems wrong. The typical way to know if an object is namespaced is to use the discovery information, would that be the right way to solve that?
They could use discovery, or can continue to use the presence of the attribute, special-casing requests (namespaces are the only cluster-scoped resource which can have the attribute populated in AdmissionReview requests).
the documentation should be clearer, and I'd consider updating the POST-based create behavior to include this information to be consistent with update/patch/delete requests
do you want to update the doc and/or the post-based create behavior?
Sure! I am happy to take care of both.
where do you think it makes sense to add the namespace to the post-based behavior? We could add it to , but that might interfere with authorization, right? Would it be better to add the namespace directly to the ?
The info from the request body is not available when parsing requestinfo, only data in the URL. That's sort of what I expected, around the point we do this to pull the name from the incoming object:
Makes sense, I will give it a go, thank you!
that will also require sweeping admission plugins that check the namespace attribute. examples:
I think that we don't need to change anymore now. Do you know if there are other places in the docs where I should check?
yeah, I think we're good now. thanks for helping identify and clean this up


Question: pod := createDNSPod(f.Namespace.Name, wheezyProbeCmd, jessieProbeCmd, dnsTestPodHostName, dnsTestServiceName) pod.Spec.DNSPolicy = v1.DNSClusterFirst pod.Spec.DNSConfig = &v1.PodDNSConfig{ <del> Nameservers: []string{kubednsIP}, Searches:    testSearchPaths, </del> <ins> Searches: testSearchPaths, </ins> Options: []v1.PodDNSConfigOption{ { Name:  "ndots",

an E2E test that assumes that a service named "kube-dns" exists in the "kube-system" namespace. This is a kubeadm-ism, and the test fails if the cluster DNS is not configured the way kubeadm configures it. Although this E2E test isn't a conformance test, making it functional on non-kubeadm clusters would increase its utility for distributors that may wish to run this E2E test as a conformance test. [sig-network] DNS HostNetwork should resolve DNS of partial qualified names for services on hostNetwork pods with dnsPolicy: ClusterFirstWithHostNet Since we ran it on an OpenShift cluster. No response The test makes assumptions about how the cluster DNS is configured. The test uses (not ), so it does not need to specify , as far as I can tell. Omitting obviates the need to look up the service. I believe that the service lookup was left in as an oversight during the development of the test when it was changed from specifying "None" to specifying "ClusterFirst": Cc: /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
responded on the pr.
It seems kubeadm inherited the status-quo, I can't find any place that mandates the Service name for DNS Asking for double checking, but the docs seems to mention that this is more a recommendation than a preescription If it seems that the test can be done in a more portable way we should do it


Question: "github.com/golang/glog" ) <ins> const statusUpdateRetries = 3 </ins> // controllerKind contains the schema.GroupVersionKind for this controller type. var controllerKind = batch.SchemeGroupVersion.WithKind("Job")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: for a Job does not work on Kubernetes 1.10.0 What you expected to happen: can limit the number of time a pod is restarted when running inside a job How to reproduce it (as minimally and precisely as possible): Use this resource file: If the job was created in Kubernetes 1.9, it will soon fail: While creating the job in Kubernetes 1.10, it will be restarted infinitely: Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.0 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Kubeadm Others:
/sig scheduling
/sig job /sig lifecycle /sig apps
/sig testing
We just updated to 1.10 and seem to be affected by this, too.
I had to add activeDeadlineSeconds to all Jobs and Hooks, as workaround...
Can confirm with 1.10.1
It seems to be introduced by cc
Cause appears to be that the method gets exactly one attempt to update the status with the failed condition and if it fails due to resource conflict, it never adds the condition. The controller code seems to use requeues to track failures, but if a sync loop executes successfully, it would lose track of the failed item ever having been requeued. Looking into a fix now.
Looking at , we force an immediate resync when we see a pod that has failed which also clears the key in our queue, but that in turn makes us lose state of the number of requeues.
This bug killed my test cluster over the long weekend (with + pods). Luckily we don't yet use CronJobs in prod.
killed my cluster too :( still existing in kubernetes 1.10.2
This issue also can't be mitigated as terminated are not part of the resource quota:
This is a pretty terrible bug and it definitely still exists in v1.10.2
/priority important-soon
can you please take a look?
Looking right now...
I've opened to address the issue. Sorry for the troubles y'all.
pulled the patch into current 1.10.2 release and it works! Thanks :)
I filed (with and a test for Job with backoffLimit=1).

The test exposes some other problems. Here are two failure mode: never retries (): the code uses to see the number of retries. However, the job can fail to be synced just because of an update conflict (). That shouldn't be seen as a retry. retries more than needed (): Job controller creates more pods than needed because it doesn't realize itself has failed and exceeded backoffLimit earlier. This may be due to stale cache or that the job status failed to be updated earlier. I think Job backoffLimit is still broken.
FWIW,(assuming I'm not missing something) I can confirm on GKE 1.10.2-gke.3 (latest as of Jun 2nd) the is still broken. The following job spec endlessly creates pods:
we used the latest v1.10.3 release but backoffLimit is still broken
This issue prevents us from rolling 1.10 to all clusters
/milestone v1.11 Looks like we have a couple of PR(s) in progress and feels like we should get this fixed by the time we ship 1.11, please let me know if this fine or not.
I think that this issue should be fixed in 1.10 regardless of 1.11 as this issue causes serious cluster stability, for a fairly common scenario. Specifically for us, migration to 1.11 adds complexity and risk due to the change in the way nvidia GPUs are managed, so it will take us some time to upgrade to 1.11.
y, guess my question is ... is it fixed in master? (and hence already fixed for 1.11), if not, do we need to get this fixed in master/1.11 and backport to 1.10
Unless I'm mistaken, the objects are considered stable APIs right? (since they are under the group). If so, I also think this should be fixed in the next 1.10 point release. In general, like we would also rather to be able to use s without being forced to upgrade to 1.11.
This is a regression in functionality, and when the proper fix is determined, it should be cherry picked back to 1.10. We need it fixed in master first, however. I'm going to escalate this to sig-apps to see if we can get the right resources.

cc: This will need to merge into master, and then be cherry picked back to 1.10
PTAL
The fix is ready for review and will be cherry-picked for the next 1.10 release.
Reopening for cherry pick.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I didn't see anything in the Changelog for v1.10.4, so i'm guessing this still hasn't been fixed yet?
The fix has landed to master but not to 1.10 yet. See It will get hopefully merged before the next 1.10.x release.
Having the same issue. Waiting for update.
This fix has been cherry picked back to 1.10, and should show up in the next patch release (1.10.5).
Any ETA for 1.10.5 yet? Even a ballpark date would be fine.
I'm planning to cut 1.10.5 tomorrow, assuming all tests will be green. In case of any last minute problems it may slip by a day or two, but so far everything looks good for tomorrow.
Thankyou to all involved! I really appreciate this being rolled back into 1.10.
Given that the fix is already on 1.11 and on 1.10 branches, this issue should be closed. Thanks for fixing this!
/close
I don't believe this is fixed? Unless I'm doing something wrong.. Both resource specs have a and . This is in GKE,
I have the same bug on 1.9, with an openshift cluster: Any plan on backporting it to 1.9 as well?
you likely are encountering a different issue, as the root cause of this bug was introduced in kubernetes 1.10, not 1.9.

Also, kubernetes 1.9 is now outside of the support window, so no fixes will be backported to the upstream branch.
I also find this bug in kubernetes 1.12.1 Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} In pod failure condition,when i set parallelism 2, backofflimit 0,it will destory the last failure pod,only left one pod remain;And set parallelism 2, backofflimit 1,sometimes it keeps 3 pods,sometimes it kills the last failure pod remains 2
@ kubectl version: Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} minikube version: v0.30.0 apiVersion: batch/v1 kind: Job metadata: name: whalesay1 spec: template: spec: containers: name: whalesay image: docker/whalesay command: ["cowsa", "This is a Kubernetes Job!"] restartPolicy: Never backoffLimit: 2 NAME READY STATUS RESTARTS AGE pod/whalesay1-6lt4b 0/1 ContainerCannotRun 0 59s

pod/whalesay1-86tlz 0/1 ContainerCannotRun 0 48s pod/whalesay1-8v8nr 0/1 ContainerCannotRun 0 1m pod/whalesay1-8w9k2 0/1 ContainerCannotRun 0 1m pod/whalesay1-8xxtr 0/1 ContainerCannotRun 0 53s pod/whalesay1-dl4l8 0/1 ContainerCannotRun 0 1m pod/whalesay1-fhlhg 0/1 ContainerCannotRun 0 37s pod/whalesay1-fz8mv 0/1 ContainerCannotRun 0 26s pod/whalesay1-g2mt6 0/1 ContainerCannotRun 0 9s pod/whalesay1-hgj95 0/1 ContainerCannotRun 0 1m pod/whalesay1-hrsrd 0/1 ContainerCannotRun 0 1m pod/whalesay1-hzm42 0/1 ContainerCannotRun 0 1m pod/whalesay1-mvctw 0/1 ContainerCannotRun 0 1m pod/whalesay1-n7zq9 0/1 ContainerCannotRun 0 2m pod/whalesay1-nl89l 0/1 ContainerCannotRun 0 31s pod/whalesay1-nqcmg 0/1 ContainerCreating 0 2s pod/whalesay1-p85rm 0/1 ContainerCannotRun 0 2m pod/whalesay1-pjzmd 0/1 ContainerCannotRun 0 2m pod/whalesay1-q2k7p 0/1 ContainerCannotRun 0 43s pod/whalesay1-rhvz7 0/1 ContainerCannotRun 0 1m pod/whalesay1-rzrtg 0/1 ContainerCannotRun 0 1m pod/whalesay1-xp5tr 0/1 ContainerCannotRun 0 1m pod/whalesay1-zn2s6 0/1 ContainerCannotRun 0 1m pod/whalesay1-zq4g9 0/1

ContainerCannotRun 0 14s pod/whalesay1-ztxdv 0/1 ContainerCannotRun 0 19s NAME DESIRED SUCCESSFUL AGE 1 0 2m
Just update to something newer than 1.10.0. That version is ages old and I'm sure it is fixed within the latest 1.10.x release. (Which is also out of support so update to 1.12/1.13 ASAP.)
In version "v1.10.11" the though set to 4 for a Job's pod, the pod eviction happens for 5 times and at few times it was for 3 times. Also to note that we also have applied and Below is the version details of Kubernetes that we are using
Though we have set the backoffLimit of a Job to 4 the pod eviction is still being done more than 4 times i.e. 5 times at one of the instances its three times as well. The eviction of the pod happens for a geniune reason i.e. the disk space of the pod node exhausts when the application within the pod executes, we know the reason why that's happening. The point is that the Job's-Pod should be attempted 4 times and not 5 times. Please note that strangely we don't get to see this error in another environment where we have v1.10.05. providing the version details of that environment below There the pod eviction happens as per the stated literature of Eviction with BackOffLimit. We have no issues there. Now when we have upgraded to v1.10.11 recently [ About two days back ] we are getting the reported problem. Would appreciate if someone in this forum can advise on the root cause and a solution. Do we have any flaw with version "v1.10.5" like security loophole.
10.11 is required to avoid a critical security bug. I imagine we will need to update to fix this, but we will need to find out which of the patched versions has the fix we need. We can then roll this through the environments as a staged update.
As noted above 1.11+ are supported by the community until 1.14 is out:
I experienced the same behavior with . Can somebody confirm that this not happening with = versions?

We are experiencing the same issue with version 1.11.9-docker-1. If it makes any sense we have 2 enviroments. One is Ubuntu 18.04.2 LTS and the other is CentOS Linux release 7.5.1804. It happens only at the Ubuntu enviroment.
In meantime we updated cluster: We still experiencing the issue.
Hi everyone! This issue is for a specific problem in Kubernetes 1.10. Right now, the upstream Kubernetes project supports the following three versions: 1.13, 1.14, 1.15. If you are experiencing a similar issue under any of those three versions, then please open a new issue and provide all the details requested. As the specific bug in this issue is resolved and on an unsupported version, I'm going to lock this closed issue. Thanks!


Question: } // Returns a PVC definition based on the namespace. <ins> // Note: if this PVC is intended to be pre-bound to a PV, whose name is not //   known until the PV is instantiated, then the func createPVPVC will add //   pvc.Spec.VolumeName to this claim. </ins> func makePersistentVolumeClaim(ns string) *api.PersistentVolumeClaim { // Specs are expected to match this test's PersistentVolume

Context: NodeSwap feature gate was recently . Currently, as per , swap is only supported with cgroup v2 - cgroup v1 is no longer supported. For this reason, we decided to fail kubelet if NodeSwap feature gate is enabled and the node runs with cgroup v1. This would help prevent any surprises for the user by failing as soon as possible. This effort is being implemented in this PR: The problem: Currently, swap jobs are running with cgroup v1. Swap tests were in a way that they first check on which environment they're being run (e.g. which cgroup version? is NodeSwap FG enabled? Which swapBehavior is defined? etc) and then expect the right behavior to take place. With v1, the tests expect the containers to be defined without swap access. While the tests pass, they are much less valuable, since all they check is that swap is disabled for v1 workloads. The tests would be much more valuable if they would run with v2, therefore actually ensure that proper swap limitations are configured. In addition, is dependent on these changes, since as of now the jobs immediately fail with this PR since kubelet fails on an environment with cgroup v1 and swap enabled. What needs to change: [x] CRI-O swap related jobs should be defined to run with cgroup v2 (e.g. ) [x] containerd swap related jobs should be defined to run with cgroup v2 (e.g. ). Future work: In the future, we might consider introducing more jobs to check swap. Ideally, both and would be tested, perhaps by periodic jobs.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/sig testing
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: APIVersion: "v1", FieldPath:  "spec.containers[3]", }, <del> Reason:  "Stopped", Message: "some verbose message: 1", Source:  v1.EventSource{Component: "eventTest"}, Count:   1, Type:    v1.EventTypeNormal, </del> <ins> Reason:              "Stopped", Message:             "some verbose message: 1", Source:              v1.EventSource{Component: "eventTest"}, ReportingController: "eventTest", Count:               1, Type:                v1.EventTypeNormal, </ins> }, expectLog:    `Event(v1.ObjectReference{Kind:"Pod", Namespace:"baz", Name:"foo", UID:"differentUid", APIVersion:"v1", ResourceVersion:"", FieldPath:"spec.containers[3]"}): type: 'Normal' reason: 'Stopped' some verbose message: 1`, expectUpdate: false,

From the Deprecated API Migration Guide: , since v1.19, the has been using the new reportingController and reportingInstance to replace the decrecated source.component and But when I checked the Kube events in a cluster with version 1.26.0, those new fields are empty and the supposed to be deprecated fields have not been renamed. Seeing the values in the new fields like reportingController and reportingInstance an Kubernetes cluster with version 1.26.0 some Kube events the Kube events details for the field of reportingController, reportingInstance, source.component, No response <detailsServer Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-12-11T00:15:47Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"} </details<detailsAzure </details<details</details<details</details<details</details<details</details
/sig instrumentation
I have also encountered this issue in my local 1.26.0 Kubernetes cluster, which does not align with the deprecation guide described in I believe I can investigate this further
/assign
Thank you for making the PR to fix this! Also a comment in the PR for the doc update.
/triage accepted


Question: cnt-- // if device is no longer used, see if need to logout the target if cnt == 0 { <del> // strip -lun- from device path ind := strings.LastIndex(device, "-lun-") prefix := device[:(ind - 1)] </del> <ins> device, prefix, err := extractDeviceAndPrefix(mntPath) if err != nil { return err } </ins> refCount, err := getDevicePrefixRefCount(c.mounter, prefix) if err == nil && refCount == 0 { // this portal/iqn are no longer referenced, log out // extract portal and iqn from device path <del> ind1 := strings.LastIndex(device, "-iscsi-") portal := device[(len("/dev/disk/by-path/ip-")):ind1] iqn := device[ind1+len("-iscsi-") : ind] </del> <ins> portal, iqn, err := extractPortalAndIqn(device) if err != nil { return err } </ins> glog.Infof("iscsi: log out target %s iqn %s", portal, iqn) out, err := c.plugin.execCommand("iscsiadm", []string{"-m", "node", "-p", portal, "-T", iqn, "--logout"}) if err != nil {

During iscsi volume cleanup, kubelet reads from and retrieves a list of device paths where iscsi volume mounts on. The device path was expected to be in the form of . But currently the device path in is in the form of , thus iscsi panicked . I suspect this panic started to happen after switched to use from . seems to always resolve device path (see output below), and thus the expected iscsi configuration in the device path no longer exists.


Question: err:        "", }, { <ins> name:       "allow create of my node pulling name from object", podsGetter: noExistingPods, attributes: admission.NewAttributesRecord(mynodeObj, nil, nodeKind, mynodeObj.Namespace, "", nodeResource, "", admission.Create, mynode), err:        "", }, { </ins> name:       "allow update of my node", podsGetter: existingPods, attributes: admission.NewAttributesRecord(mynodeObj, mynodeObj, nodeKind, mynodeObj.Namespace, mynodeObj.Name, nodeResource, "", admission.Update, mynode),

Is this a BUG REPORT or FEATURE REQUEST? (choose one):BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-alpha.4.52+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T06:57:30Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-alpha.4.52+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T06:57:30Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know: None


Question: continue } <del> // Check the node readiness condition (logging all). for i, cond := range node.Status.Conditions { Logf("Node %s condition %d/%d: type: %v, status: %v", name, i+1, len(node.Status.Conditions), cond.Type, cond.Status) // Ensure that the condition type is readiness and the status // matches as desired. if cond.Type == api.NodeReady && (cond.Status == api.ConditionTrue) == wantReady { Logf("Successfully found node %s readiness to be %t", name, wantReady) return true } </del> <ins> if isNodeReadySetAsExpected(node, wantReady) { return true </ins> } } Logf("Node %s didn't reach desired readiness (%t) within %v", name, wantReady, timeout) return false } <ins> // Filters nodes in NodeList in place, removing nodes that do not // satisfy the given condition // TODO: consider merging with pkg/client/cache.NodeLister func filterNodes(nodeList *api.NodeList, fn func(node api.Node) bool) { var l []api.Node for _, node := range nodeList.Items { if fn(node) { l = append(l, node) } } nodeList.Items = l } </ins> // LatencyMetrics stores data about request latency at a given quantile // broken down by verb (e.g. GET, PUT, LIST) and resource (e.g. pods, services). type LatencyMetric struct {

I ran e2e tests against docker 1.7.0-rc2 release, found a bunch of issues. Here is one I think it belongs to kubernetes for sure. 1) ResizeNode test failed, the machine is removed from instance group: But kubectl get nodes still returns 3 ndoes: This might be caused by the new docker binary I am testing against here. This is not the issue I am filing here. 2) Now networking test is running later, I noticed that the test is putting a pod on each node: nettest-gwtc4 on node e2e-test-dawnchen-minion-5l11 I haven't checked networking code in detail yet, not sure the issue is test itself, or scheduler issue. cc/ since you recently changed this. cc/
The networking test doesn't use the scheduler, it manually fills in the host field.
Ok, then we should filter out NotReady nodes from testing; otherwise, the test itself has the dependencies on other tests.
Yes, that test is doing a questionable thing.
Agreed.
I implemented skipping not-ready nodes in . Is it enough to close this issue, or is there something else to be done here?
I think that should fix the issue. ?
status: waiting for merge of


Question: }, }, }, <ins> { name: "spec:v6 ip:v6 dualstack:off isIPv6Primary:true", isIPv6Primary:          true, specExpctPrimary:       true, clusterIPExpectPrimary: true, enableDualStack:        false, svc: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo", ResourceVersion: "1"}, Spec: api.ServiceSpec{ Selector:  map[string]string{"bar": "baz"}, Type:      api.ServiceTypeClusterIP, IPFamily:  &ipv6Service, ClusterIP: "2000::1", }, }, }, { name: "spec:v6 ip:v6 dualstack:on isIPv6Primary:true", isIPv6Primary:          true, specExpctPrimary:       true, clusterIPExpectPrimary: true, enableDualStack:        true, svc: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo", ResourceVersion: "1"}, Spec: api.ServiceSpec{ Selector:  map[string]string{"bar": "baz"}, Type:      api.ServiceTypeClusterIP, IPFamily:  &ipv6Service, ClusterIP: "2000::1", }, }, }, { name: "spec:v6 ip:v4 dualstack:on isIPv6Primary:true", isIPv6Primary:          true, specExpctPrimary:       true, clusterIPExpectPrimary: false, enableDualStack:        true, svc: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo", ResourceVersion: "1"}, Spec: api.ServiceSpec{ Selector:  map[string]string{"bar": "baz"}, Type:      api.ServiceTypeClusterIP, IPFamily:  &ipv6Service, ClusterIP: "10.0.0.1", }, }, }, { name: "spec:v4 ip:v4 dualstack:on isIPv6Primary:true", isIPv6Primary:          true, specExpctPrimary:       false, clusterIPExpectPrimary: false, enableDualStack:        true, svc: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo", ResourceVersion: "1"}, Spec: api.ServiceSpec{ Selector:  map[string]string{"bar": "baz"}, Type:      api.ServiceTypeClusterIP, IPFamily:  &ipv4Service, ClusterIP: "10.0.0.1", }, }, }, { name: "spec:v4 ip:v6 dualstack:on isIPv6Primary:true", isIPv6Primary:          true, specExpctPrimary:       false, clusterIPExpectPrimary: true, enableDualStack:        true, svc: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo", ResourceVersion: "1"}, Spec: api.ServiceSpec{ Selector:  map[string]string{"bar": "baz"}, Type:      api.ServiceTypeClusterIP, IPFamily:  &ipv4Service, ClusterIP: "2000::1", }, }, }, </ins> } for _, tc := range testCases {

What happened: When K8s is started in dual-stack mode with IPv6 as the "main" (or base) family services specified with ipFamily:IPv4 still get a ipv6 ; The is expicitly set to "10.0.0.0" using . What you expected to happen: Services with ipFamily:IPv4 shall get ipv4 ClusterIP addresses. How to reproduce it (as minimally and precisely as possible): Setup a K8s cluster in dual-stack mode with family IPv6 as base Define a service with ipFamily: IPv4 To make K8s think IPv6 is the main family the cue is to define ipv6 addresses to K8s binaries. Below are my settings as an example. I do not use but install "from scratch"; Master processes; is accessed with ipv4 which does not affect the K8s family. Worker processes; Anything else we need to know?: There are other problems with dual-stack/ipv6-base but may be caused by this one. For example does not work and some processes does not become "ready". Ipv6 traffic to ipv6 services and DNS works; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: - OS (e.g: ): BusyBox based own setup. Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): Others:
/sig network /cc
The problem seem to be; The problem occurs when an ipv6 range is specified first, even in an ipv4-based dual-stack cluster. But the ipv6 range must be specified first otherwise the "kubernetes" service gets an ipv4 address but an ipv6 backend; ~Perhaps the way to fix this problem is to make sure the service address is from the right family and require that the order to is "ipv4,ipv6".~ Bad idea!
There seem to be a option; But I can't see it in the help printout. I will try to use it...
:disappointed:
Here it where the kubernetes service address is set;
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: return nil } } <ins> // If we're past the position of the last describe's rparen, pop the describe off if lastDescribe > -1 && node != nil { if node.Pos() > v.describes[lastDescribe].rparen { v.describes = v.describes[:lastDescribe] } } </ins> return v }

What happened: tried promoting a test with a tag in its full test description. The hierarchy of ginko-relevant AST's was . The tag was a in the top level . The full test description only came from , so the tag was missed. What you expected to happen: The conformance test list generator should have logged and ERROR and failed a test on the PR, calling out Disruptive as an invalid tag How to reproduce it (as minimally and precisely as possible): Check out locally and follow the directions in to generate the conformance list. The invalid test is in the list. Anything else we need to know?: Is there a function we can use to get the full test description from Ginkgo instead of rolling our own? If not, we should probably be maintaining a stack rather than just the last FYI who ran into this FYI who wrote the code to keep track of last
/sig architecture /area conformance
/milestone v1.16
/priority important-soon
/assign
GitHub didn't allow me to assign the following users: devaips. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: tags = ["automanaged"], ) <ins> go_test( name = "k8s.io/apiserver/pkg/endpoints/metrics_test", srcs = ["k8s.io/apiserver/pkg/endpoints/metrics/metrics_test.go"], library = ":k8s.io/apiserver/pkg/endpoints/metrics", tags = ["automanaged"], ) </ins> filegroup( name = "package-srcs", srcs = glob(["**"]),

Some of the exported prometheus metrics, e.g. include high-cardinality fields like the User-Agent. This creats a large number of time-series on the prometheus server is ususally not advised to do. Please consider reducing the cardinality of the exported metrics.


Question: skipPlugins.Insert(pl.Name()) continue } <del> metrics.PluginEvaluationTotal.WithLabelValues(pl.Name(), metrics.PreFilter, f.profileName).Inc() </del> if !s.IsSuccess() { s.SetFailedPlugin(pl.Name()) if s.IsUnschedulable() {

We observed degradation in scheduling throughput, putting it under 100 pods/s. Respective pprof of kube-scheduler points it to ! /sig scheduling /kind regression 100 pods/node n/a No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/sig scalability
Thanks for noticing it /assign
just to confirm this bug is present only in 1.27 ( introduced with pluginevaluationtotal metric) , isn't it ?
Exactly, 1.27+ (so the current master as well) to be specific.


Question: framework.ExpectNoError(err, "3 pods not found") // delete Collection of pods with a label in the current namespace <del> err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).DeleteCollection(context.TODO(), metav1.DeleteOptions{}, metav1.ListOptions{ </del> <ins> err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).DeleteCollection(context.TODO(), metav1.DeleteOptions{GracePeriodSeconds: &zero}, metav1.ListOptions{ </ins> LabelSelector: "type=Testing"}) framework.ExpectNoError(err, "failed to delete collection of pods")

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce (5% of runs) pull-kubernetes-e2e-kind-ipv6 pull-kubernetes-e2e-gce-ubuntu-containerd Which test(s) are flaking: This is the flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
merged 2 weeks ago in
/assign
potential mitigation in
To track down the test flaking there was some great feedback in by and was created as a potential fix based on that feedback /cc


Question: return pdbs[0] } <ins> // This function returns pods using the PodDisruptionBudget object. // IMPORTANT NOTE : the returned pods should NOT be modified. </ins> func (dc *DisruptionController) getPodsForPdb(pdb *policy.PodDisruptionBudget) ([]*v1.Pod, error) { sel, err := metav1.LabelSelectorAsSelector(pdb.Spec.Selector) if sel.Empty() {

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No. What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): DisruptionController getPodsForPdb Is this a BUG REPORT or FEATURE REQUEST? FEATURE REQUEST in we have function return pods using . but at the end we perform a copy of the pod array that we listed with : this is not necessary cc
/assign
/kind enhancement


Question: if sp.TargetPort.Type == intstr.Type(intstr.Int) { w.Write(LEVEL_0, "TargetPort:t%d/%sn", sp.TargetPort.IntVal, sp.Protocol) } else { <del> w.Write(LEVEL_0, "TargetPort:t%d/%sn", sp.TargetPort.StrVal, sp.Protocol) </del> <ins> w.Write(LEVEL_0, "TargetPort:t%s/%sn", sp.TargetPort.StrVal, sp.Protocol) </ins> } if sp.NodePort != 0 { w.Write(LEVEL_0, "NodePort:t%st%d/%sn", name, sp.NodePort, sp.Protocol)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: when using , the TargetPort is wrong What you expected to happen: if the targetPort is string, show the name correctly. Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
could you paste more details about the Kubernetes Version and the yaml you used to create the service? I tried to reproduce this error but it worked fine in my environment.
use the master branch. yaml output
I'm not sure if the targetPort could be set to string type, looks like the targetPort here makes no sense of port1 for it doesn't point to an actual port. I think we have to validate this value rather than displaying the string. WDYT?
It can be a string, you can see this
thanks for your ref, I will try to fix this and pls let me know if any new progress :)
I have commit a pr
ok, thanks~


Question: if check_cm_response == 0: # There is an existing ConfigMap, patch it <del> patch = '{"data":{"max-body-size":"1024m"}}' </del> <ins> patch = '{"data":{"body-size":"1024m"}}' </ins> patch_cm_command = kubectl + ['patch', 'cm', cm_name, '-p', patch] patch_cm_response = call(patch_cm_command)

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): nginx-load-balancer-conf Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): ` Environment: Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS Kernel (e.g. ): 4.4.0-71-generic -Ubuntu SMP Fri Mar 24 12:59:01 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: juju Others: What happened: I used the "registry" action to create docker registry inside the cluster. It modified nginx-load-balancer-conf config map. It replaced: body-size: 1024m with (incorrect): max-body-size: 1024m What you expected to happen: Expected ConfigMap to stay as it was How to reproduce it (as minimally and precisely as possible): Deploy cluster with juju and then use "registry" action to create docker registry Anything else we need to know:


Question: return } if shouldHide(&version, selfVersion) { <del> klog.Warningf("This metric(%s) has been deprecated for more than one release, hiding.", d.fqName) </del> <ins> // TODO(RainbowMango): Remove this log temporarily. https://github.com/kubernetes/kubernetes/issues/85369 // klog.Warningf("This metric(%s) has been deprecated for more than one release, hiding.", d.fqName) </ins> d.isHidden = true } })

cc who found it. the reason is the calls from the import here: i suspect the problem surfaced with this bug fix. cc more refs: is an anti-pattern in this case. ideally metrics registration should not trigger warnings for a simple version output. mitigation on the user side is to pipe stderr to a null device (as long as klog is writing to stderr) /kind bug /priority important-longterm /area kubelet /wg component-standard
/sig node
should be fixed before 1.17.0, doesn't block beta.2
cc
Could this be caused by the following in init() of ? It was marked I wonder when deprecatedRequestLatency can be dropped.
looks like was deprecated in 1.14 - Is that the only one that needs to be dropped? are there more.
Dropping restclientrequestlatencyseconds should remove the warning for --version. In the 1.14 changelog, there were quite a few other metrics deprecated. I guess we don't have to drop them now.
/assign It's an interesting issue. I will investigate it and get back to you. By the way: I tried this morning, I got 4 warnings, things getting worse as we deprecated more metrics this weekend. (the log can not point the exact name of a metric, we intended to improve it in v1.18)
it's being invoked from here: the message is not helpful, since it doesn't tell you which metric is deprecated
Yes. It's a known issue and we intended to improve it in v1.18. Now, I suggest removing the log temporarily, Fired a PR , for reasons as follows: Too many binaries have been affected.(kubelet, kube-proxy, kube-apiserver, kube-controller-manager, kube-proxy, kube-scheduler). We should consider more about this issue. v1.17.0 is fast approaching, we should make a quick fix. All components try to register a deprecated metrics will be affected. Maybe we should record deprecated/hidden metrics by a new metric instead of the log.
Summary the root cause here: According to metrics deprecated policy, if a metric's - = 1, it will become a hidden metric. Hidden means we will not register it to the Prometheus registry by default.

If we try to register a hidden metric to the stability framework, the warning log will be printed out. Because some metrics registered by init() calls from the import, so...
I have tried to print the metric's name at local: kubelet: kube-proxy: kube-apiserver: kube-controller-manager: kube-scheduler:
ideally the registration should happen outside of init(), which means calling it on demand. but i will defer to the owning sigs.
In principle, yes, I agree. Otherwise, we need to be very very careful with every log in the stability framework. I think we can track it and work out by iteration. cc for comments.


Question: if az.serviceOwnsRule(service, *existingRule.Name) { keepRule := false klog.V(10).Infof("reconcileLoadBalancer for service (%s)(%t): lb rule(%s) - considering evicting", serviceName, wantLb, *existingRule.Name) <del> if findRule(expectedRules, existingRule) { </del> <ins> if findRule(expectedRules, existingRule, wantLb) { </ins> klog.V(10).Infof("reconcileLoadBalancer for service (%s)(%t): lb rule(%s) - keeping", serviceName, wantLb, *existingRule.Name) keepRule = true }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the Idle Timeout annotation was set to "3" which isn't a supported case, kube-controller-manager is stuck in deleting Azure LB. New service with the same name couldn't be created again even without the annotation. What you expected to happen: New service should be created successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug
/assign Since I'm seeing this on some of our clusters
/priority important-soon
Thanks
- The problem here seems that when we want to call (when processing a delete event) it reconciles the loadbalancer which in turn tries to get the idleTimeout, resulting in an error. one of the strategies is to still construct our diff in case of a delete and exercise the code path at the end. This basically translate into ignoring the error when getting idleTimeout and excluding it from finding if the rule is dirty or not when we are . toughts ? Generally speaking, whenever we update a loadbalancer with unsupported values on annotations, ensuring load balancer will fail resulting in an un-cleaned LB, this is a cross-provider issue. /sig cloud-provider cc
cc
lgtm. the delete operation should continue to do cleanup job even some wrong options are provided from users.


Question: } // When enforcing quotas are enabled, we'll condition this // on their being disabled also. <del> if ibytes > 0 { ibytes = -1 </del> <ins> fsbytes := ibytes if fsbytes > 0 { fsbytes = -1 </ins> } <del> if err = setQuotaOnDir(path, id, ibytes); err == nil { </del> <ins> if err = setQuotaOnDir(path, id, fsbytes); err == nil { </ins> quotaPodMap[id] = internalPodUid quotaSizeMap[id] = ibytes podQuotaMap[internalPodUid] = id

When sizeLimit is specified subsequent invocations will fail with the message: No error should be raised. Create a pod with emptyDir and sizeLimit with FSQuotaMonitoring enabled, MountVolume.SetUp is called again only if the first iteration takes longer then the actual reconciler loop interval (100ms). No response <detailsversion.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-beta.0.87+", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-04-10T09:05:56Z", GoVersion:"go1.20.2", Compiler:"gc", Platform:"linux/amd64"} </details<detailsSelf-managed cluster in Azure </details<detailsNAME="Red Hat Enterprise Linux" VERSION="9.1 (Plow)" ID="rhel" IDLIKE="fedora" VERSIONID="9.1" Linux test-master-0 6.3.0-rc3 SMP PREEMPTDYNAMIC Thu Apr 6 20:44:15 UTC 2023 x8664 x8664 x8664 GNU/Linux </details<detailsContainerd 1.6.15 </details<detailsCNI: CANAL 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
May you please share more information? Which version of Kubernetes you are using? OS version etc.
The problem occurs with the latest master branch, I updated the initial post with versions.
Thank you.


Question: } } <ins> func expectStop(t *testing.T, w *worker, c bool, msg string) { if c { t.Errorf("[%s - %s] Expected to stop, but did not", w.probeType, msg) } } </ins> func resultsManager(m *manager, probeType probeType) results.Manager { switch probeType { case readiness:

What happened: liveness/readiness probe fails while pod is terminated. Also it happened only once during the pod termination. The issue started happening after upgrading version to v1.7 from v1.6.X How to reproduce it (as minimally and precisely as possible): execute to delete pod, so status of the nginx-podA1 is changed to , right after that it seems Liveness and Readiness Probe is executed and failed, but only once. Nginx reverse proxy is running in the pod. so I just use method for liveness and readiness Here is my Deployment config. Here is Events log by Environment: Kubernetes version: 1.7.2
/kind bug
/sig node
PING
Since the upgrade to 1.7, it seems our deployment rollouts have a higher failure rate. Occasionally, the pod would come up, but no readiness probe ever gets started. It stays in that state, blocking the entire deployment. I usually have to delete the pod so it is rescheduled, and a new readiness probe is fired to check. I wonder if these are related issues here.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen /remove-lifecycle rotten We see this consistently with all pods who define a liveness or readiness probe. Whenever we roll out a new deployment, the pods who are terminated will emit a failed liveness/readiness probe AFTER they have been terminated.

We have considered adding a hook that just sleeps for 2-3 seconds, but it seems like a band-aid solution to something that should not happen in the first place. Is this an impossible-to-solve race condition between kubernetes moving parts?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Do you still experience this? This happens to us running kubernetes 1.11.4.
On k8s 1.12.7 and this is happening to me as well.
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} still happen. I think just add a phase "Terminateing" in pod, when an pod is in Terminateing just stop check.
I think it still happening in 1.13, why this issue was closed? Logs, !
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
still experience this on kubernetes 1.12.4. delete event and probe failed event so it seems that after the pod is deleted, the probe keeps running.I am wondering if the probe is started before the pod deletion.
/area kubelet
/priority important-soon
Any progress on this issue? It seems like some kind bug.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /assign
GitHub didn't allow me to assign the following users: ashleyschuett. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /remove-lifecycle rotten /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
16.9 here. Same issue.
/assign
I am also facing this issue. We have pods which have a lot of cleanup to do during shutdown, it can take up to 5 mins to terminate gracefully. During this time the livelinessProbe is detecting failure and restarting the pod. not really what we want. I am unable to prevent the service that handles the liveliness check from stopping while the cleanup is happening. It would be better if the pod was immediately removed from the service and the probes stopped while the shutdown is performed. Basically this ends up that k8s never actually is able to terminate the pod.
I will take that point and propose a shutdown probe to API sig.

Same here at 1.18.8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
kubernetes_version 1.19.3. Same issue.
Same here, kubernetes 1.19.4.
/remove-lifecycle rotten
/assign
Should I consider a cherry-pick for 1.20 and 1.19? (maybe 1.18 too?)
we're running 1.16 and being hit by this continuously when some of our elixir apps shutdown so cherry-picking it in 1.18 would at least put it closer in our update path :angel:


Question: DetachDiskByName(diskName, diskURI string, nodeName types.NodeName) error // GetDataDisks gets a list of data disks attached to the node. GetDataDisks(nodeName types.NodeName) ([]compute.DataDisk, error) <ins> // GetProvisioningStateByNodeName gets the provisioning state by node name. GetProvisioningStateByNodeName(name string) (string, error) </ins> }

Implement proposal for Azure We should implement interface in Azure cloud provider. returns true if the instance is in stopped state to detach volumes. But if instance is deleted we should not return error. Related features: Related proposal: Related PRs: /kind feature /sig azure /cc
- can you assign me on this one ? I already have a PR in flight.
That's great! /assign just loop in for this RP review, thanks.
GitHub didn't allow me to assign the following users: yastij. Note that only and repo collaborators can be assigned. <detailsIn response to : That's great! /assign just loop in for this RP review, thanks. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
o, you are not a k8s member yet, anyway, you could send out the PR for this feature, related to this issue, thanks.
Thanks, looking forward to your PR.


Question: name = "go_default_library", srcs = [ "options.go", <ins> "plugins.go", </ins> "validation.go", ], deps = [ "//pkg/api:go_default_library", "//pkg/api/validation:go_default_library", <ins> "//pkg/cloudprovider/providers:go_default_library", </ins> "//pkg/features:go_default_library", "//pkg/kubeapiserver/options:go_default_library", "//pkg/kubelet/client:go_default_library", "//pkg/master/ports:go_default_library", <ins> "//plugin/pkg/admission/admit:go_default_library", "//plugin/pkg/admission/alwayspullimages:go_default_library", "//plugin/pkg/admission/antiaffinity:go_default_library", "//plugin/pkg/admission/defaulttolerationseconds:go_default_library", "//plugin/pkg/admission/deny:go_default_library", "//plugin/pkg/admission/exec:go_default_library", "//plugin/pkg/admission/gc:go_default_library", "//plugin/pkg/admission/imagepolicy:go_default_library", "//plugin/pkg/admission/initialization:go_default_library", "//plugin/pkg/admission/initialresources:go_default_library", "//plugin/pkg/admission/limitranger:go_default_library", "//plugin/pkg/admission/namespace/autoprovision:go_default_library", "//plugin/pkg/admission/namespace/exists:go_default_library", "//plugin/pkg/admission/noderestriction:go_default_library", "//plugin/pkg/admission/persistentvolume/label:go_default_library", "//plugin/pkg/admission/podnodeselector:go_default_library", "//plugin/pkg/admission/podpreset:go_default_library", "//plugin/pkg/admission/podtolerationrestriction:go_default_library", "//plugin/pkg/admission/priority:go_default_library", "//plugin/pkg/admission/resourcequota:go_default_library", "//plugin/pkg/admission/security/podsecuritypolicy:go_default_library", "//plugin/pkg/admission/securitycontext/scdeny:go_default_library", "//plugin/pkg/admission/serviceaccount:go_default_library", "//plugin/pkg/admission/storageclass/setdefault:go_default_library", "//plugin/pkg/admission/webhook:go_default_library", </ins> "//vendor/github.com/spf13/pflag:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/net:go_default_library", <ins> "//vendor/k8s.io/apiserver/pkg/admission:go_default_library", </ins> "//vendor/k8s.io/apiserver/pkg/server/options:go_default_library", "//vendor/k8s.io/apiserver/pkg/storage/storagebackend:go_default_library", ],

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The current help message from kube-apiserver is wrong. It shows only the 'NamespaceLifecycle' plug-in is available, and the default value is . Users cannot know what are the available plug-ins, and they get confused where comes from. What you expected to happen: The correct output should be this: How to reproduce it (as minimally and precisely as possible): Run using master or at least 1.7.2. Anything else we need to know?: This is a regression introduced when refactoring code into generic server code. Environment: Kubernetes version (use ): 1.7.2+ ? Cloud provider or hardware configuration**: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Any Others: None
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, it is really exist. If no guys works on this, i will pick it up.
Thanks for confirming this. I'm already working on it. Will submit pr in minutes.
cool. :)


Question: if err := podPassesBasicChecks(pod, g.pvcLister); err != nil { return result, err } <ins> trace.Step("Basic checks done") </ins> // Run "prefilter" plugins. preFilterStatus := g.framework.RunPreFilterPlugins(state, pod) if !preFilterStatus.IsSuccess() { return result, preFilterStatus.AsError() } <ins> trace.Step("Running prefilter plugins done") </ins> numNodes := g.cache.NodeTree().NumNodes() if numNodes == 0 {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We have enabled in genericscheduler. However, the steps being traced seem outdated. In particular, the "snapshot" is expensive and should be traced. Other steps can be traced as well wherever appropriate, e.g., PreFilter. Why is this needed: Improve granularity of tracing. /sig scheduling
/assign


Question: local binaries binaries=($(kube::golang::binaries_from_targets "${targets[@]}")) <del> local platform for platform in "${platforms[@]}"; do kube::golang::set_platform_envs "${platform}" kube::log::status "Building go targets for ${platform}:" "${targets[@]}" local -a statics=() local -a nonstatics=() for binary in "${binaries[@]}"; do if kube::golang::is_statically_linked_library "${binary}"; then kube::golang::exit_if_stdlib_not_installed; statics+=($binary) else nonstatics+=($binary) fi done </del> <ins> local parallel=false if [[ ${#platforms[@]} -gt 1 ]]; then local gigs gigs=$(kube::golang::get_physmem) </ins> <del> if [[ -n ${use_go_build:-} ]]; then # Try and replicate the native binary placement of go install without # calling go install.  This means we have to iterate each binary. local output_path="${KUBE_GOPATH}/bin" if [[ $platform != $host_platform ]]; then output_path="${output_path}/${platform////_}" fi for binary in "${binaries[@]}"; do local bin=$(basename "${binary}") if [[ ${GOOS} == "windows" ]]; then bin="${bin}.exe" fi if kube::golang::is_statically_linked_library "${binary}"; then kube::golang::exit_if_stdlib_not_installed; CGO_ENABLED=0 go build -installsuffix cgo -o "${output_path}/${bin}"  "${goflags[@]:+${goflags[@]}}"  -ldflags "${version_ldflags}"  "${binary}" else go build -o "${output_path}/${bin}"  "${goflags[@]:+${goflags[@]}}"  -ldflags "${version_ldflags}"  "${binary}" fi done </del> <ins> if [[ ${gigs} -gt ${KUBE_PARALLEL_BUILD_MEMORY} ]]; then kube::log::status "Multiple platforms requested and available ${gigs}G > threshold ${KUBE_PARALLEL_BUILD_MEMORY}G, building platforms in parallel" parallel=true </ins> else <del> # Use go install. if [[ "${#nonstatics[@]}" != 0 ]]; then go install "${goflags[@]:+${goflags[@]}}"  -ldflags "${version_ldflags}"  "${nonstatics[@]:+${nonstatics[@]}}" fi if [[ "${#statics[@]}" != 0 ]]; then CGO_ENABLED=0 go install -installsuffix cgo "${goflags[@]:+${goflags[@]}}"  -ldflags "${version_ldflags}"  "${statics[@]:+${statics[@]}}" fi </del> <ins> kube::log::status "Multiple platforms requested, but available ${gigs}G < threshold ${KUBE_PARALLEL_BUILD_MEMORY}G, building platforms in serial" parallel=false </ins> fi <del> done </del> <ins> fi if [[ "${parallel}" == "true" ]]; then kube::log::status "Building go targets for ${platforms[@]} in parallel (output will appear in a burst when complete):" "${targets[@]}" local platform for platform in "${platforms[@]}"; do ( kube::golang::set_platform_envs "${platform}" kube::log::status "${platform}: go build started" kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-} kube::log::status "${platform}: go build finished" ) &> "/tmp//${platform////_}.build" & done local fails=0 for job in $(jobs -p); do wait ${job} || let "fails+=1" done for platform in "${platforms[@]}"; do cat "/tmp//${platform////_}.build" done exit ${fails} else for platform in "${platforms[@]}"; do kube::log::status "Building go targets for ${platform}:" "${targets[@]}" kube::golang::set_platform_envs "${platform}" kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-} done fi </ins> ) }

Filing for This code is wrong The local needs to be on a separate line from the command execution, and we need to be resilient to not existing on OS X, like:
For once I was actually watching the output of a build and noticed that I was seeing an error about md5sum not being found. I'm building on a mac using boot2docker. The short_hash function () works fine on my mac as md5 does exist.
Actually, I'm just going to fix this as a follow-on to .
Thank you!