Question: } if chosen == "" { <del> glog.Warningf("Could not assign a mount device (all in use?).  mappings=%v, valid=%v", self.deviceMappings, valid) </del> <ins> glog.Warningf("Could not assign a mount device (all in use?).  mappings=%v, valid=%v", deviceMappings, valid) </ins> return "", false, nil } <del> self.deviceMappings[chosen] = volumeID </del> <ins> self.attaching[chosen] = volumeID </ins> glog.V(2).Infof("Assigned mount device %s -> volume %s", chosen, volumeID) return chosen, false, nil } <del> func (self *awsInstance) releaseMountDevice(volumeID string, mountDevice mountDevice) { </del> <ins> func (self *awsInstance) endAttaching(volumeID string, mountDevice mountDevice) { </ins> self.mutex.Lock() defer self.mutex.Unlock() <del> existingVolumeID, found := self.deviceMappings[mountDevice] </del> <ins> existingVolumeID, found := self.attaching[mountDevice] </ins> if !found { <del> glog.Errorf("releaseMountDevice on non-allocated device") </del> <ins> glog.Errorf("endAttaching on non-allocated device") </ins> return } if volumeID != existingVolumeID { <del> glog.Errorf("releaseMountDevice on device assigned to different volume") </del> <ins> glog.Errorf("endAttaching on device assigned to different volume") </ins> return } glog.V(2).Infof("Releasing mount device mapping: %s -> volume %s", mountDevice, volumeID) <del> delete(self.deviceMappings, mountDevice) </del> <ins> delete(self.attaching, mountDevice) </ins> } type awsDisk struct {

Environment: CoreOS 815.0.0, k8s 1.1.0 () It happens to me quite often that a volume marked as "available" on AWS remains in a "detached" state according to the kubelet: Deleting the pod and recreating it eventually succeeds.
I have some fixes in progress that should fix this
/cc
I can reliably reproduce similar issue, I am not sure it's the same: create a AWS EBS, let's say it has ID Create a pod that uses this EBS: Create the pod. It succeeds eventually: Delete the pod Create the pod again. It will be forever IMO, it's caused by not removing the detached volume from internal map of attached volumes . Subsequent will find it there and it won't even try to attach it again. Patch is available in referenced PR.
can you please confirm that your bugs is the same as I can reproduce? Or do you see different bug, only with similar symptoms?
It is
Was this fixed by ?
I think you referenced the wrong issue / PR, didn't you? As far as I know this is not fixed yet.
Yes, I meant to link . mentioned there that it might fix this issue.
Oh right! It made it to v1.2.0-alpha.4
did not fix this issue (but fixed the DeviceMapping cache invalidation)
I think, what is happening is that you run into an error or timeout while waiting for the disk operation. That leaves the devicecache yet again in an inconsistent state. I have a fix here that just discards the cache on any error. Additionally, it is totally possible that there is concurrent operations which lead to an error and bork the cache. The fix: Unfortunately, this is in our tests still not 100% rock solid. Especially in timeout cases. I'm missing the capability to properly test this to engage in a bigger refactoring of this whole issue. A workaround to actually fixing it properly would be to add a TTL to the cache, so that it eventually starts working again.
Just for information for people who might use the GitHub search, as of Kubernetes 1.1.7 the errors logged by the look like: Deleting the pod or restarting doesn't help, the instance keeps failing attaching the EBS with the same errors if the pod get rescheduled on the same host.

thanks for the fix!
Oh... the same host. I bet I messed up that case... I'm going to move into 1.2 so I take another look at this!
What's the status on this?
I just don't think this is fixable with the volume cache we have. I proposed a PR to remove the cache, and replace it with a map of active attachment operations in progress. A little less efficient, but a lot more correct (presuming passes testing).
How to check what K8 version has this fix?
Is there any progress on this issue? I'm running K8s version v1.5.2, and I'm having similar issues. awsElasticBlockStore volumes are virtually unusable due to one of three errors I'm getting: My volume claim is ReadWriteOnce, so this next error confuses me: And finally: [SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected.] Unable to mount volumes for pod "staging-connect-mysql-<id(<id)": timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql] Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql]
Also having the same issue as ^ Should we be using PV+VPC instead of claiming the EBS volume directly from the deployment spec?


Question: TestSampleAPIServer(f, aggrclient, imageutils.GetE2EImage(imageutils.APIServer)) }) <del> ginkgo.It("should manage the lifecycle of an APIService", func() { </del> <ins> /* Release: v1.25 Testname: Aggregator, manage lifecycle of an APIService Description: An APIService is created which MUST succeed. The APIService status when replaced MUST succeed. Given the updating of the APIService status, the fields MUST equal the new values. The APIService status when patched MUST succeed. Given the patching of the APIService status, the fields MUST equal the new values. The APIService when replaced MUST succeed. Given the updating of the APIService, the fields MUST equal the new values. It MUST succeed at deleting a collection of APIServices via a label selector. */ framework.ConformanceIt("should manage the lifecycle of an APIService", func() { </ins> ns := f.Namespace.Name framework.Logf("ns: %v", ns)

[X] Test approval issue: [X] Test PR: 7 July 2022 [x] Two weeks soak end date: 21 Jul 2022 [x] Test promotion According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. replaceApiregistrationV1APIServiceStatuspatchApiregistrationV1APIServiceStatusreplaceApiregistrationV1APIServicedeleteApiregistrationV1CollectionAPIService Using a number of existing e2e test practices a new has been created for 4 APIService endpoints. The e2e logs for this test are listed below. This query shows the following endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: b.errs = append(b.errs, fmt.Errorf("no unstructured mapper provided")) return b } <del> if b.mapper != nil { </del> <ins> if b.mapper != nil && b.mapper != b.unstructured { </ins> b.errs = append(b.errs, fmt.Errorf("another mapper was already selected, cannot use unstructured types")) return b }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I run and I got Running with shows What you expected to happen: How to reproduce it (as minimally and precisely as possible): With Kubernetes master branch and in different terminal on the same machine Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: VM via libvirtd OS (e.g. from /etc/os-release): Fedora 27 with golang-1.10.1-1.fc29.x86_64 from rawhide Kernel (e.g. ):Install tools: ?? Others:
/sig cli
/assign
/assign


Question: REGION=$(grep -o "^[a-z]*-[a-z0-9]*" "${KUBE_TEMP}"/cluster-location.txt) MASTER_NAME="${KUBE_GCE_INSTANCE_PREFIX}"-master <del> MASTER_INTERNAL_IP=$(gcloud compute addresses describe "${MASTER_NAME}-internal-ip"  --project "${PROJECT}" --region "${REGION}" -q --format='value(address)') </del> <ins> if [[ ${GCE_PRIVATE_CLUSTER:-} == "true" ]]; then MASTER_INTERNAL_IP=$(gcloud compute addresses describe "${MASTER_NAME}-internal-ip"  --project "${PROJECT}" --region "${REGION}" -q --format='value(address)') fi </ins> MASTER_IP=$(gcloud compute addresses describe "${MASTER_NAME}-ip"  --project "${PROJECT}" --region "${REGION}" -q --format='value(address)')

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubemark cluster fails to be created when KUBEGCEPRIVATE_CLUSTER is set to false. What you expected to happen: Kubemark cluster should be created. How to reproduce it (as minimally and precisely as possible): Running ./start- script. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing /sig scalability
/assign
IIUC the bug was introduced in The problem is with not being guarded with checking which causes the error:


Question: "k8s.io/kubernetes/pkg/util/goroutinemap" vol "k8s.io/kubernetes/pkg/volume" "k8s.io/kubernetes/pkg/volume/csimigration" <ins> "k8s.io/kubernetes/pkg/volume/util" </ins> "k8s.io/klog/v2" )

What happened: With a StorageClass with , it's not clear why a PVC is not bound when there is no available PV. On Pod, there is a clear event what's wrong: On the PVC side we get only this event: Note that the message says "waiting for first consumer'. But the pod already exists. The reason why it's not bound yet is that the StorageClass has no dynamic provisioning configured () and there are no free PVs to satisfy the claim. What you expected to happen: An event "1 node(s) didn't find available persistent volumes to bind" on the PVC.
/sig scheduling /sig storage
I tried to fix this in VolumeBinding predicate, The predicate is called for each pod - node combination, so it would generate event for each node. Which is too much and it's even wrong when there is a node that has an available PV. We need to evaluate all nodes and then, if the pod is still unschedulable, distribute events to all unbound PVCs of that pod. Scheduler does not offer any way how to track unbound PVCs at that stage - whole volume logic is hidden in VolumeBinding predicate. One option would be to extend  struct to contain also list of objects where to send events (except the pod itself): Scheduler would then send all failed scheduling events it sends to a pod to all the , without any interpretation. is used to manage duplicates easily - each object (its UID) can be in the map only once. Simple array would work too, but then it needs extra work in removing duplicates. Not sure how easy it is to detect duplicate instances. is extensible to allow any object type. predicate would put all unbound PVCs there. does not need any special handling of each PVC. Message "2 node(s) didn't find available persistent volumes to bind" sounds just right in case a pod uses multiple PVCs, each of them might be bind-able on different nodes (node A may satisfy PVC1, but not PVC2, while node B may satisfy PVC2 only). what do you think about such design?
We used to have extension point, which I believe is the right moment for recording the event for unbounds PVCs. But recently it's been refactored so doesn't fit. Do you think this case can be a good example of using as an informational extension point?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale wdyt?
I have some other concerns. What if the pod is deleted? Should the scheduler send events that PVC is not in volume binding? The PV controller must be updated to stop emit event in this scenario. It might be simpler to solve on the PV controller side. In this function, we can check if the PVC is being used by a pod. If it is, we can send event like , then people can go to check the progress of pod "foo/bar". If the pod "foo/bar" is deleted, the PV controller can emit "waiting for first consumer ..." event again. What do you think? cc
Sounds better than it's now. My only issue may be performance - PV controller has to check if a pod exists, i.e. list all pods in the same namespace as PVC being reconciled. I am probably too worried.
current attach detach controller has a custom indexer to index pods by PVC keys to avoid iterating all pods every time to find pods which reference given PVC. And there are other places we need to check if a PVC is used by a pod or not from pod informer, e.g. pvcprotectioncontroller pvcontroller we can try to share this custom indexer among these volume controllers in the future. This will improve the overall performance of the IMO. What do you think
Yes, using the indexer looks like a good idea.
/remove-sig scheduling I also agree that solving in PVC side seems like a better idea. Thanks
/assign unbound claims are reconciled in the PV controller every 15 seconds (default), we may have bad worst-case performance (e.g. too many pods in the same namespace). we can add an indexer in the PV controller locally to solve this issue first (also optimize ), then find a way to in manager to improve the overall performance.


Question: <del> package(default_visibility = ["//visibility:public"]) load( "@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library", ) go_binary( name = "cauldron", embed = [":go_default_library"], ) go_library( name = "go_default_library", srcs = ["cauldron.go"], importpath = "k8s.io/kubernetes/test/soak/cauldron", deps = [ "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library", "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/rest:go_default_library", "//test/e2e/framework:go_default_library", "//vendor/k8s.io/klog:go_default_library", ], ) filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [":package-srcs"], tags = ["automanaged"], ) </del>

We've deleted all repos from Docker Hub and will be retiring the org/user (ref: ) Apparently the repo is referenced for docker push and pull in It's unclear to me whether we have broken anything, or if this is dead code. I would use to find out if this is referenced anywhere (aside from sundry vendor dirs) If dead code: remove it If used: move to test/e2e/images and follow the same pattern there used to push to /sig testing /area test /wg k8s-infra /priority important-soon /help /good-first-issue I believe this is a pretty straightforward issue and am happy to review whatever PR's are made for it
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : We've deleted all repos from Docker Hub and will be retiring the org/user (ref: ) /area test /wg k8s-infra /priority important-soon /good-first-issue I believe this is a pretty straightforward issue and am happy to review whatever PR's are made for it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nil } <ins> // do the SMB mount with username, password, remotepath // return (output, error) func newSMBMapping(username, password, remotepath string) (string, error) { if username == "" || password == "" || remotepath == "" { return "", fmt.Errorf("invalid parameter(username: %s, password: %s, remoteapth: %s)", username, password, remotepath) } // use PowerShell Environment Variables to store user input string to prevent command line injection // https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment_variables?view=powershell-5.1 cmdLine := `$PWord = ConvertTo-SecureString -String $Env:smbpassword -AsPlainText -Force` + `;$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $Env:smbuser, $PWord` + `;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Credential $Credential` cmd := exec.Command("powershell", "/c", cmdLine) cmd.Env = append(os.Environ(), fmt.Sprintf("smbuser=%s", username), fmt.Sprintf("smbpassword=%s", password), fmt.Sprintf("smbremotepath=%s", remotepath)) output, err := cmd.CombinedOutput() return string(output), err } // check whether remotepath is already mounted func isSMBMappingExist(remotepath string) bool { cmd := exec.Command("powershell", "/c", `Get-SmbGlobalMapping -RemotePath $Env:smbremotepath`) cmd.Env = append(os.Environ(), fmt.Sprintf("smbremotepath=%s", remotepath)) _, err := cmd.CombinedOutput() return err == nil } // remove SMB mapping func removeSMBMapping(remotepath string) (string, error) { cmd := exec.Command("powershell", "/c", `Remove-SmbGlobalMapping -RemotePath $Env:smbremotepath -Force`) cmd.Env = append(os.Environ(), fmt.Sprintf("smbremotepath=%s", remotepath)) output, err := cmd.CombinedOutput() return string(output), err } </ins> // Unmount unmounts the target. func (mounter *Mounter) Unmount(target string) error { klog.V(4).Infof("azureMount: Unmount target (%q)", target)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: azure file remount on Windows in same node would fail What you expected to happen: If user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since command would fail if file share is already mounted on the node: It's related to bug: Error logs would be like following: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9-v1.12, master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign
/sig windows
/sig azure
/milestone v1.14 cc This may be good to backport to 1.12 or 1.13 too if customers are using those versions.


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cache import ( "fmt" "testing" "time" apierrors "k8s.io/apimachinery/pkg/api/errors" testingclock "k8s.io/utils/clock/testing" ) type retryScenarioStep struct { clockStep time.Duration err       error wantRetry bool } func TestRetryWithDeadline(t *testing.T) { internalError := apierrors.NewInternalError(fmt.Errorf("etcdserver: no leader")) otherError := fmt.Errorf("some other error") testCases := []struct { name        string duration    time.Duration reset       time.Duration isRetryable func(error) bool scenario    []retryScenarioStep }{ { name:        "Never retry when duration is zero", duration:    time.Duration(0), reset:       time.Second * 30, isRetryable: func(err error) bool { return false }, scenario: []retryScenarioStep{ { clockStep: time.Second * 1, err:       nil, wantRetry: false, }, { clockStep: time.Second * 0, err:       internalError, wantRetry: false, }, { clockStep: time.Second * 1, err:       otherError, wantRetry: false, }, }, }, { name:        "Retry when internal error happens only within duration", duration:    time.Second * 1, reset:       time.Second * 30, isRetryable: apierrors.IsInternalError, scenario: []retryScenarioStep{ { clockStep: time.Second * 1, err:       internalError, wantRetry: true, }, { clockStep: time.Second * 1, err:       internalError, wantRetry: true, }, { clockStep: time.Second * 1, err:       internalError, wantRetry: false, }, }, }, { name:        "Don't retry when other error happens", duration:    time.Second * 1, reset:       time.Second * 30, isRetryable: apierrors.IsInternalError, scenario: []retryScenarioStep{ { clockStep: time.Second * 1, err:       otherError, wantRetry: false, }, }, }, { name:        "Ignore other errors for retries", duration:    time.Second * 1, reset:       time.Second * 30, isRetryable: apierrors.IsInternalError, scenario: []retryScenarioStep{ { clockStep: time.Second * 1, err:       internalError, wantRetry: true, }, { clockStep: time.Second * 0, err:       otherError, wantRetry: true, }, { clockStep: time.Second * 1, err:       internalError, wantRetry: true, }, }, }, } for _, tc := range testCases { fakeClock := testingclock.NewFakeClock(time.Now()) retry := NewRetryWithDeadline(tc.duration, tc.reset, tc.isRetryable, fakeClock) for i, step := range tc.scenario { fakeClock.Step(step.clockStep) retry.After(step.err) result := retry.ShouldRetry() if result != step.wantRetry { t.Errorf("%v unexpected retry, step %d, result %v want %v", tc, i, result, step.wantRetry) break } } } } </ins>

When etcd is unable to elect leader for 3 election cycles, etcd returns "no leader" error through watches. This can be seen in api-server logs as . Also, when apiserver receives this error, it terminates all watchers: . In large clusters, this can destabilize master, as all these watchers need to be recreated almost at the same time, overwhelming api-server quite often. I propose adding new api-server flag that allows to specify timeout, during which we are retrying to reestablish apiserver <-etcd watches, without terminating all api-server watchers. If we are unable to reestablish watches within this specified timeout, we start terminating all watchers. Losing etcd leader for ~2-3s does not destabilize api-server. Trigger prolonged reelection of etcd leader. No response Observed in all kubernetes versions 1.19-1.24 N/A <details</details<details</details<details</details<details</details
/sig scalability /sig api-machinery /priority important-soon
/wg reliability
/assign
/triage accepted
indeed, this is a source of instability. Is this coming from ?
I believe it's coming from this change. As I mentioned, the idea to mitigate it is to add some parameter that will allow us to ignore this error coming from etcd for specific duration without terminating all watchers, for example 10s be default.
With this comment thread in mind: It would probably be cleaner to allow for simply setting the "timeout" as part of WithRequireLeader option (instead of using the constant of 3). But that would require etcd changes, including the API and we want to workaround that in much smaller timeframe. But a comment about how to make that cleaner with some additional context from here would help.


Question: // Implement Redirector. func (storage *SimpleRESTStorage) ResourceLocation(ctx api.Context, id string) (string, error) { // validate that the namespace context on the request matches the expected input <del> requestedResourceNamespace := api.Namespace(ctx) if storage.expectedResourceNamespace != requestedResourceNamespace { return "", fmt.Errorf("Expected request namespace %s, but got namespace %s", storage.expectedResourceNamespace, requestedResourceNamespace) </del> <ins> storage.requestedResourceNamespace = api.Namespace(ctx) if storage.expectedResourceNamespace != storage.requestedResourceNamespace { return "", fmt.Errorf("Expected request namespace %s, but got namespace %s", storage.expectedResourceNamespace, storage.requestedResourceNamespace) </ins> } storage.requestedResourceLocationID = id if err := storage.errors["resourceLocation"]; err != nil {

is always using the cross namespace context. This needs to be fixed to support scoped watches.


Question: runningPods = []*v1.Pod{} for _, pod := range podList.Items { <del> if r, err := testutils.PodRunningReady(&pod); err != nil || !r { </del> <ins> if r, err := testutils.PodRunningReadyOrSucceeded(&pod); err != nil || !r { </ins> continue } runningPods = append(runningPods, &pod)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Restart: Always pods went into OutOfCpu error state after kubelet restart. Pods should remain running as node had sufficient capacity for them. Completed, RestartNever pod should not revert from Completed to OutOfCpu. Launch kubelet with: Run the following script against the local cluster: <details</detailsObserve pod status: Restart the kubelet: Observe pods: xref /sig node Kubernetes version (use ): v1.23.0-alpha.3.183+ - Cloud provider or hardware configuration: Local machine OS (e.g: ) & Kernel (e.g. ):Install tools: local-up- Network plugin and version (if this is a network-related bug): N/A Others:
/triage accepted /priority critical-urgent Reproducible on 1.22
From the reproducer: yields However, in the kubelet logs, it sees 7100m cpu allocated?
Looks like the topology admit handler thinks that the completed pods are using CPU which is why when it gets to the 8th it fails it with OutOfCpu. Must be a bug in there somewhere.
/assign
Unclear if could be related but linking for posterity
Confirmed this only affects 1.22, repro on 1.21 has expected behaviour: /sig scheduling I'll start a bisect.
is
Added some logging With this we see I think that the terminated pods are getting filtered, but for whatever reason they are still getting their resource requests to NodeInfo?
points out that my logging language was a bit backwards and that the completed pods should not have been returned, because they already successfully completed and exited. This code is suspect: The logs show that the completed pods are never filtered, so they are failing one of these checks. I am going to try to write a test case that hits this issue so we can test fixes against it.

The completed pods definitely show up as complete, but for some reason they do not get filtered (note I have changed my patch above for the log lines to make sense now): So I guess isPodTerminationRequested=true on already exited pods, but I would only expect that to be true if the pod was actively in the process of termination and not already exited. cc Log patch: <details</details
I tried changing to but apparently it thinks that's true :( Edit: doesn't work either.
The bug is that the only component in the kubelet that is allowed to authoritatively state a container is terminated (pod worker) takes non-zero time to do so during startup (because we have to call kill container to guarantee it's stopped). This races with the pod config loop and admission, which sends all the containers at once. In order for the race to be eliminated, some equivalent of a sync has to be completed during startup on terminal but potentially non-terminated pod workers BEFORE other adds are processed. That could be running HandlePodCleanups once all sources are synced but before HandlePod* methods are called, but HandlePodAdditions() cannot be invoked until after the pod worker has had a chance to verify all terminal pods are actually terminal. EDIT: Another potential approach is for the pod worker to hold starting pods until SyncKnownPods observes all terminal pods are accounted for at least once, then allowing starts.
Also, we appear to be missing a kubelet restart test that verifies admission with completed pods (which I thought was on our list of "missing e2e tests that would have caught these regressions" already). Such a test would have to: pods that consume resources for admission until they are running some of them with failed and` successful states the kubelet some of the running pods the kubelet none of the terminal pods are restarted, all running pods are restarted, and that only non-terminal pods are accounted to admission


Question: } } <del> expectedRVs := []string{"0", "20", "", "30", ""} </del> <ins> expectedRVs := []string{"0", "20", "", "30", "", "40", ""} </ins> if !reflect.DeepEqual(listCallRVs, expectedRVs) { t.Errorf("Expected series of list calls with resource version of %#v but got: %#v", expectedRVs, listCallRVs) }

What happened: Using client-go v1.18.8 to watch resources on API server versions 1.17.x and 1.18.0 - 1.18.5 After the connection to the server is temporarily interrupted the client never recovers from error See the earlier issue for steps to reproduce: This was assumed fixed by But it appears the fix for the client was (probably inadvertently) implemented in a non-backward-compatible fashion and only solves the issue when the server is running v1.18.6 or later as well.
/sig api-machinery
Cc
PR
We generally recommend having apiservers be updated first before all other components. But yeah - it's true that this doesn't work for clients being more fresh - let me take a look at your PR.
/cc
This will be addressed by (and the cherrypicking it back).
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This has been fixed by and corresponding cherrypicks.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I have disconnected a node from network for a few minutes. After reconnecting, I keep receiving such error messages from kubelet on the node, even after 15 minutes in reconnected state: What you expected to happen: I expect that after network recovery kubelet reconnects to the Apiserver as before, and after a recovery such timeouts do not occur. How to reproduce it (as minimally and precisely as possible): Just have a node connected to the cluster. Then, disconnect it from the network for 3-4 minutes, then reconnect. Then observe kubelet's logs. Anything else we need to know?: I have strict tcp keepalive settings in place on master and worker nodes, but this should not be the cause. Restarting kubelet solves the issue, the error messages disappear. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: bare metal OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): kube-router Others:
/sig api-machinery
/sig api-machinery
It seems that after some more time, kubelet recovers. Sometimes it takes one hour, sometimes two. Recovery is successful when I receive similar logs: What is the reason that it takes so long for it to recover?
The root cause is , deduplicating against that issue
Debugging, tracing the code, I am not really sure that this is the same issue as the other mentioned here. Error messages show that the client is requesting higher resource versions that is available at the server: , the former is greater by 989 than the latter. I dont know from where this resource version comes from. I've made a very stupid patch on my kubelet/client-go: I am pretty sure that this is not the best solution, but, it seems that it forces a full list every time ListAndWatch is called.
You're correct, sorry about that... I misread the timeout message. The error it is getting is coming from the server, which means kubelet -apiserver connectivity is fine.

This message comes from the watch cache, which means the watch cache in the API server is behind where the kubelet's informer was, which is pretty surprising. A few questions: Is there a single apiserver or multi-apiserver cluster? Is there a single etcd member or multiple etcd members? Is there an API server running on the node that was detached/reattached?
cc does this ring any bells?
cc for eyes on watch cache
I don't think I've hit this problem specifically. Since this could happen due to an etcd partition or due to a partition between a apiserver and etcd, the answers questions asked in seem like the appropriate next things to figure out.
Yeah - I agree with Joe. It seems like some network partition issue of the apiserver the node was connected to.
I will be able to answer the questions in detail later, but in short: there are 3 masters set up with kubeadm, haproxy load-balances the queries to the apiservers. The only thing to reproduce the issue is to disconnect the node from the network for a few minutes. Masters or etcd members are not touched, and anyway the cluster seems to be healthy.
Today, it occured on a diffrent cluster. Setup is similar: stacked etcd on master nodes, haproxy load-balances traffic between them. Just was playing with haproxy servers and the HA setup, and after a few restarts of haproxy, one of the nodes entered this state. So during a haproxy restart, all kubelets' connections to apiservers are terminated, but rarely, one of them enters this unhealthy state.
Multi apiserver cluster Stacked etcd members are present. I think In my setup it is not relevant, as all the kubelets are talking to the apiserver through the load-balancer, which picks one of the available apiservers. I am doing a rolling restarts of the nodes, and there were at least one occurence, when even one of the master nodes got stucked in this state. How can I help debugging/fixing this?
How can we proceed with this issue?

got same problem in a old cluster, ( up since 3+years) after updating to kube 1.18.3 , one node kubelet start logging those errors ( ~10 errors/minute with various resources) I've now just restarted kubelet and now seems working again.. Update: while testing I've tried to connect the kubelet of that node to each single master nodes to check if there were some kind of network issue / partitioning.. none of the master servers returned errors ( the kubelet worked fine with all master nodes bypassing the nginx proxy that manage masters HA
I think this problem is getting serious. I am not 100% sure, but today, during node restarts (including masters) coredns was migrated multiple times, and unfortunately, kube-proxy pods had not received the updated situation, thus, dns resolution was not working inside the cluster. After detecting this, a simple restart of all kube-proxy pods resolved our problem. There were no network issues during this, so I suspect we've hit the same bug, now in kube-proxy.
I am trying to dig this. Now, I've written a little go prog which uses the same reflector what kubelet uses. It just starts a watch for CSIDrivers, and I've made client-go to print the last resourceVersion. Starting the program multiple times (i.e. connecting to different masters) produces the following: Right now I dont exactly know what url is being fetched and what arguments are passed in, I still have to figure that out. But now it seems, that different masters return different resouceVersions. Etcd does not report any problems/issues.
Any ideas, suggestions where to look further?
Today I have re-initialized 2 of my 3 master nodes to make sure etcd replicas are not corrupted. Just after the nodes have been joined, the same behavior could be observed: connecting to different masters resulted in different resourceVersions.
Now, it can be seen, that querying different apiservers again return different resourceVersions: 192.168.8.60: is a haproxy load-balancing traffic between 3 masters.
please let me know where to go further. My cluster runs on PI boards, perhaps, not the fastest ones.

But if this issue depends on slow hardware, than it can pop up anytime on fast hardware as well.As I wrote, today I reinitialized 2 of my 3 masters to make sure no partitioning occured. Also, please have a look at my comments. Or, if you have a clue that my setup is broken, please let me know!
It seems that it happens to resources where no objects exist at all. Can it be?
Ahh - the fact tht it happens for types with no objects actually made me know what is happening here. Basically what is happening is that when kube-apiserver is starting, watchcache is initialized from "now" (i.e. it will be synced to resource version that was globally last in etcd at the time when it was initialized). However, later it's updated only on changes of resources of those types. So when no objects of that type is being /modified/deleted it will stay the same. So it's kind-of "working-as-implemented" from the kube-apiserver perspective. That can be improved - I'm actually already looking into reusing etcd progress-notify feature to get rid of this behavior ( & ) - I would like to have alpha for that in 1.20 and beta in 1.21 (planning to write down KEP once I'm back from my soon-starting vacation), but that shouldn't be the problem by itself. And it's not what is really biting you. The real problem is that there is a bug in reflector in: It handled the "too old RV" error, but it doesn't handle "too large RV" correctly. What it should be doing instead is to: trigger relist from etcd (to get RV=now") start watching from that RV Watch will work fine because watch will simply wait until events with larger RV appear. I will send out a PR fixing that bug (will try to do it later today).
is out for review
This has been fixed/mitigated in 1.18. Going to backport to 1.18 now. It seems 1.18 is the first version affected (the offending changes were reverted in 1.17 months ago).
This has been mitigated in 1.18 and head. The more proper fix is proposed in Closing this one.

Hello, as i saw, i have the same issue in 1.19.3 version: Our setup is 3 etcd, 2 apiserver bound on those 3 etcd and a haproxy in front of apiservers
If that happens once - that can definitely happen (and that's fine). The bug was that the components were stuck with those errors.
18.0 get same error . restart docker and kubelet return OK. so , should I update the 1.18.0 to 1.18.6+ ?


Question: if isManagedDisk { managedDisk := &compute.ManagedDiskParameters{ID: &diskURI} <ins> if diskEncryptionSetID == "" { if vm.StorageProfile.OsDisk != nil && vm.StorageProfile.OsDisk.ManagedDisk != nil && vm.StorageProfile.OsDisk.ManagedDisk.DiskEncryptionSet != nil && vm.StorageProfile.OsDisk.ManagedDisk.DiskEncryptionSet.ID != nil { // set diskEncryptionSet as value of os disk by default diskEncryptionSetID = *vm.StorageProfile.OsDisk.ManagedDisk.DiskEncryptionSet.ID } } </ins> if diskEncryptionSetID != "" { managedDisk.DiskEncryptionSet = &compute.DiskEncryptionSetParameters{ID: &diskEncryptionSetID} }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure data disk should use same encryption key as os disk if key is not provided by user, otherwise user would mess up with too many keys What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: docker_pull( name = "debian-hyperkube-base-amd64", <del> digest = "sha256:d83594ecd85345144584523e7fa5388467edf5d2dfa30d0a1bcbf184cddf4a7b", </del> <ins> digest = "sha256:cc782ed16599000ca4c85d47ec6264753747ae1e77520894dca84b104a7621e2", </ins> registry = "k8s.gcr.io", repository = "debian-hyperkube-base-amd64", <del> tag = "0.9",  # ignored, but kept here for documentation </del> <ins> tag = "0.10",  # ignored, but kept here for documentation </ins> ) docker_pull(

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Cloud providers are using the command-line tool to retrieve status and format volume mounts. is missing from the hyperkube image resulting in missing volumes. Issues: What you expected to happen: Volume mounts on cloud providers are expected to be formatted and mounted. How to reproduce it (as minimally and precisely as possible): GCE and Openstack volume mounts are effected. Anything else we need to know?: Environment: Kubernetes version (use ): 1.9, 1.10+ Cloud provider or hardware configuration: GCE and Openstack OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/cc
/sig storage

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): container-os Kernel (e.g. ): Linux gke-wordpress-cluster-default-pool--m764 4.4.21+ SMP Fri Feb 17 15:34:45 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux Install tools: Others: What happened: ` I was able to bring up wordpress okay the first time, except GKE wasn't creating loadbalancer ips due to a quota issue, which I resolved, note at this point the mysql pod was up and had attached to it's volume. Upon deleting the wordpress deployment and creating it again I started getting the above errors. I deleted the mysql pod as well and brought it up again to see that it had the same issue. The volumes are backed by a gluster cluster on GCE. Looking at the brick logs on one of the gluster nodes I see I've tried restarting kubelet on the node. I can't find the kubelet log file on any of gke nodes. And I don't know how to get the kube-controller and apiserver logs from the master (GKE). I suspect it's a failing of the glusterfs client on the gke nodes? What you expected to happen: Deployment to mount the volumes successfully. How to reproduce it (as minimally and precisely as possible): Not sure how, but I've run into this intermittently Anything else we need to know: The volumes are backed by gluster
/cc
If you delete pod and create pods soon, it is possible that you could see some errors like "timeout expired waiting for volumes to attach/mount" temporally. Could you wait for a few more minutes to see whether the pods could be started correctly?
I actually tried that, deleted the pods, then waited 5 minutes and recreated them with the same error. Maybe it wasn't long enough
Can you post output of ? It may have some other events for failure..
I deleted the cluster this afternoon, but I checked constantly and those were the only errors, Timeout on mount and a failed to sync error
If you use level 4 log, it should also give the detailed messages of failed mount so that we can figure out why it failed.

Can you email me your project name, cluster name, and zone information? I can take a look at the master logs to see if there's anything useful there.
This is starting to happen more frequently..
could you provide more details about in what situation it happened frequently? Kubelet log could give us more information. If you use GKE cluster, run "journctlctl -u kubelet /tmp/log" to get the log. You could also share with us the log if it is ok.
This seems to be the relevant part of the log I re-created that and my pods sprang back up. Weird thing is I had created that, along with the pvcs and pvs but seems it's somehow getting deleted. There's almost certainly certain I'm doing wrong but I'm not sure what. ]
HI Jonathan, can you clarify what you had to recreate?
The glusterfs endpoints , Michelle Au <:
I have a very similar problem running OpenShift 3.4 in AWS EC2 doing the same exact thing. Brand new OSE install [root master]# oc version oc v3.4.1.12 kubernetes v1.4.0+ features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.31.40.126:8443 openshift v3.4.1.12 kubernetes v1.4.0+ POD in question is the docker-registry (kind of important for OSE to work) I've followed this to see if it would fix the issue: But no luck or change. Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] then: Unable to mount volumes for pod "docker-registry-3-afbbt_default(-1a45-11e7-abf8-)": timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] I have no idea what log(s) to look at, this was of no help when I deployed a new docker-registry: oc logs -f dc/docker-registry Any help or guidance would be greatly appreciated as this is a blocker issue for me.

thanks and best wishes,
are you using glusterfs for registry-storage volume or other type of volume? Could you share us your kubelet log if possible over slack Thanks!
Hi Jing Xu, Would you know where this log would be? Or is the name kublet<somethingso I may find it for you? I just requested to join the slack channel, as soon as I'm signed in I will share the log. Probably tomorrow as I'm on the east coast USA Thank you! Joel
For you are using GKE with GCI image, you can use "journalctl -u kubelet " to get the log. Otherwise, it should be in " , shepp67 <:
I also experienced this. Kubernetes version:Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Install tools: custom That kubernetes worker vm actually had volumes attached. I saw it on OpenStack dashboard. But: and actually: Besides, this is a new worker vm. I have other worker vms already running, using cinder volumes and there were no such errors. I use Helm to deploy this Pod. I have deployed it earlier and removed it. But it was 2 days ago. I found the solution here: . On that worker vm I run: Edit: or upgrade Kubernetes to = 1.6.0-beta.3 I don't know if this is the solution to issue.
I'm seeing this same error on all my manually launched private Ubuntu based clusters (6), running 1.5.3 and 1.6.0. In my case the messages reference only stopped pods launched by CronJob Jobs. The messages indicate issues mounting every volume in the pod including PVs and node-local volumes. These pods, and future ones launched by these CronJobs, are fully successful when they run. It is only after they run to successful completion and exit that these messages show up. Glancing at the kubelet code, this appears to be coming out of some housekeeping/synch loop.
I've identified a bug and am working on a fix for when "Unable to mount volumes for pod..." + "Error syncing pod" happen indefinitely BUT the pod is actually running just fine, as observed by .

For others, if you see this message but your pod is Running, you can safely ignore it. It's caused by a small limitation of the reconstruction of state (what volumes are mounted) so there is a chance you will see this only if you restart kubelet while your pods had volumes mounted. correction: you say they even Completed, they weren't just Running fine. So yeah, it's just misleading logging and not really an indication of anything breaking
Because the error message is very generic, I think there might be different problems mentioned in this thread. 's issue could be fixed by 's PR are you still have any issues with mount? could you also let us know whether you still have the issue? Thanks! Jing
Hi my issue was related to which helped me sort out. I was creating the gluster endpoints without a service and turns out the kube-controller deletes all such endpoints that are not attached to a service at regular intervals. So after every so often when a pod went down and tried to come up again and mount its respective pv it failed because it couldn't connect to gluster. The fix is to create a service for the gluster endpoints.
I also just ran into such timeout issue: After running a load test against my postgres backed web application, the database didn't respond anymore for still unknown reasons. According to kubernetes it was up and running The log showed a bunch of these messages. Not sure yet why but possibly it couldn't access the volume anymore: Short after that, the pod entered "error" state: The event log showed: This error disappeared after a few minutes (not sure how long exactly, but it took a while), but then I got the same error as described here: I couldn't figure out anything more on my own. Nothing in the syslog, nothing more than this in the kubelet log. Finally I rebooted the machine which fixed the issue.
cc do you know what error of the network not ready? It might be helpful if you could share the kubelet log with us to debug. Thanks!
AFAIK, the "network not ready error" is benign, kubelet will retry after this error () cc
I hope I didn't miss some secrets in there, quite a talky log. But here it is: The database didn't respond anymore because it ran OOM and was killed, as also shown in the log.

Although it appears like it was OOM killed after it became unresponsive in the log.. Either way, a bunch of other things got killed too, so granted - "not ideal" situation :). Yet it shouldn't drop the volume and should re-attach it.
I think you are hitting the issue mentioned above. He has a PR to fix it. I will make sure it can merge soon. Thanks!
Great, thanks! Do you know roughly when this will be available in GCE?
you mean GKE or GCE? For GKE it will take a little bit longer. On GCE, when the PR gets merged, the fix should be available when the next release is out (I think should be soon)
Running on GKE. I'd rather not launch my product with this bug around. Is there a workaround for it or something I can monitor for? Or maybe it could get hot fixed on GKE? If there is no workaround, this effectively renders GKE useless for stateful stuff.
It's hard to say if volumes are actually being dropped/unmounted or if the FailedMount errors are wrong & misleading, so I wouldn't go so far as to say it's broken because of some volume issues. It could be something else entirely. If nothing else, the PR jing's referring to will clean up the event noise and help us debug the root cause. In the meantime, I wouldn't trust the FailedMount events to be accurate, so if your applications are unresponsive, there may be other causes worth considering
Well, I only can say that my volumes were dropped and it didn't recover on it's own (only after reboot). Dropped a seen by kubernetes, dunno if the volumes were still attached but they weren't usable by my scheduled database. To be clear: I'm not "complaining" and this was just test/R&D, but it would this is the scariest kind of issue for me to happen in prod, given that it's a managed service where I can't find a workaround on my own. Rebooting fixed it in my case, but that's a big hammer..
You don't need to reboot the node. Delete the pod and restart it should work for this issue. Is this workaround ok for your case? , discordianfish <:
Good to know, thanks!

Will try this next time.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hi, I'm facing the same issue with k8s 1.9, installed by kops on AWS. volume created successfully on AWS and attached to the right node, but the pod is failed to start.
/remove-lifecycle stale
could you please provide more details about your issue? What are the steps to reproduce it? Thanks!
yeah can you open a new bug for EBS and cc me with details? I was hoping we fixed most of EBS problems in 1.9 - I will be more than happy to walk through and find what is causing volume to not mount. As I understand this is different from typical EBS problems where volume fails to attach. In your case, volume is being attached to right node but it is not being seen by kubelet.
I found the issue, and it was the fact that i used c5 instances with nvme, this is why the volume never been attached as k8s looking for /dev/xvdXX and with nvme that path is different. I did notice that in 1.9 k8s should support nvme natively but it seems that it's still not stable. thanks
oh weird. You should open a bug about NVMe support then. I was thinking it would just work.
cc Is there an issue related to nvme support? please let us know any issues you still have with 1.9. Thanks!
This is nvme on AWS, not related to the work is doing on gce.
Thanks Definitely related to NVMe support. Ran into this trying to mount a gp2 EBS volume on a node which was an m5 EC2 instance.
I think I am seeing like this with , but unsure of "why" at the moment. It would seem works perfectly fine.
did you guys had a chance to open a BUG for NVMe support on EBS? If you do - please paste output of and PersistentVolume spec.

We can't fix it unless we know bit more. :(
sorry, we worked around this by reverting back to m4 class instances. It should be simple enough to reproduce however... create a k8s cluster w/ m5 EBS backed nodes and create a simple pod w/ a persistent volume. You'll notice that the pod will never launch and you should be able to debug further. If I do run into this again I'll be sure to get some more data for you.
I too have the error, error syncing pod. kubectl describe pod kafka1-kafka-0 .... .... .... Warning FailedSync 5m (x141 over 35m) kubelet, 10.186.. Error syncing pod Warning BackOff 57s (x164 over 35m) kubelet, 10.186.. Back-off restarting failed container kubectl logs kafka1-kafka-0 -p export KAFKABROKERID=kafka1-kafka-0 export KAFKAADVERTISEDLISTENERS=PLAINTEXT://172.30.*.:9092 /etc/confluent/docker/run sh: 3: /etc/confluent/docker/run: not found


Question: Convert_unversioned_LabelSelector_to_map, Convert_Slice_string_To_Slice_int32, <ins> Convert_Slice_string_To_v1_DeletionPropagation, </ins> ) }

makes API server accepts DeleteOptions as a query parameter, but APIserver cannot parse , which is a new field in 1.6. Probably it's a bug in the parameterCodec. cc
IMO this is not a 1.6 release blocker
does this apply to all DeleteOptions fields? does a 1.5 server handle the orphanDependents field sent by correctly?
No, is handled correctly by the apiserver built from head. And for , because our generated clients still encode DeleteOptions as body, so they are not affected.
- we can close this one right?
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Why was this closed? I'm using a Kubernetes 1.9 cluster and this is still an issue. If I pass propagationPolicy as part of the query string (as per the documentation), I still receive the same error message. If I pass a DeleteOptions object with the propagationPolicy within, it works.
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
what documentation were you referring to? Is it the openapi doc? I don't know if we can just deprecate the support of query parameters for DELETE. I understand it's a breaking change, but it has been broken for a year.
Everything in delete needs to be moved to query parameters (unfortunately) as it is apparently non-standard to have a body in a delete request.

Chao Xu <:
/reopen
/remove-lifecycle rotten
cc
/assign
The DELETE operation states that propagationPolicy is a query parameter. Is correct that it's non-standard to have a body on a delete request.
GKE default version is still 1.9.7


Question: // NewDelayingQueue does not emit metrics. For use with a MetricsProvider, please use // NewDelayingQueueWithConfig instead and specify a name. // <del> // Deprecated: use TypedNewDelayingQueue instead. </del> <ins> // Deprecated: use NewTypedDelayingQueue instead. </ins> func NewDelayingQueue() DelayingInterface { return NewDelayingQueueWithConfig(DelayingQueueConfig{}) } <del> // TypedNewDelayingQueue constructs a new workqueue with delayed queuing ability. // TypedNewDelayingQueue does not emit metrics. For use with a MetricsProvider, please use // TypedNewDelayingQueueWithConfig instead and specify a name. func TypedNewDelayingQueue[T comparable]() TypedDelayingInterface[T] { </del> <ins> // NewTypedDelayingQueue constructs a new workqueue with delayed queuing ability. // NewTypedDelayingQueue does not emit metrics. For use with a MetricsProvider, please use // NewTypedDelayingQueueWithConfig instead and specify a name. func NewTypedDelayingQueue[T comparable]() TypedDelayingInterface[T] { </ins> return NewTypedDelayingQueueWithConfig(TypedDelayingQueueConfig[T]{}) } // NewDelayingQueueWithConfig constructs a new workqueue with options to // customize different properties. // <del> // Deprecated: use TypedNewDelayingQueueWithConfig instead. </del> <ins> // Deprecated: use NewTypedDelayingQueueWithConfig instead. </ins> func NewDelayingQueueWithConfig(config DelayingQueueConfig) DelayingInterface { return NewTypedDelayingQueueWithConfig[any](config) } <ins> // TypedNewDelayingQueue exists for backwards compatibility only. // // Deprecated: use NewTypedDelayingQueueWithConfig instead. func TypedNewDelayingQueue[T comparable](config TypedDelayingQueueConfig[T]) TypedDelayingInterface[T] { return NewTypedDelayingQueueWithConfig(config) } </ins> // NewTypedDelayingQueueWithConfig constructs a new workqueue with options to // customize different properties. func NewTypedDelayingQueueWithConfig[T comparable](config TypedDelayingQueueConfig[T]) TypedDelayingInterface[T] {

TypedNewDelayingQueue in does not follow the naming convention in the same package. The naming pattern observed in suggests that TypedNewDelayingQueue should be renamed to NewTypedDelayingQueue. Because we have NewTypedDelayingQueueWithConfig, and other functions are named in such rules, I think this is an obvious conversion tooling error and/or cut and paste error. There is no reproduction steps. No response <detailskubernetes v1.31.0 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
Added in /cc
/cc /triage accepted
I can change this but then that of course breaks everyone who uses the current name. Is that something we want to do?
Adding NewTypedDelayingQueue and deprecating TypedNewDelayingQueue keep backward compatibility if that is a way to go.


Question: }) }) <ins> // waitForPodStatusByInformer waits pod status change by informer func waitForPodStatusByInformer(c clientset.Interface, podNamespace, podName string, timeout time.Duration, condition func(pod *v1.Pod) (bool, error)) error { stopCh := make(chan struct{}) checkPodStatusFunc := func(pod *v1.Pod) { if ok, _ := condition(pod); ok { close(stopCh) } } controller := newInformerWatchPod(c, podNamespace, podName, checkPodStatusFunc) go controller.Run(stopCh) after := time.After(timeout) select { case <-stopCh: return nil case <-after: defer close(stopCh) return fmt.Errorf("timeout to wait pod status ready") } } // newInformerWatchPod creates a informer for given pod func newInformerWatchPod(c clientset.Interface, podNamespace, podName string, checkPodStatusFunc func(p *v1.Pod)) cache.Controller { _, controller := cache.NewInformer( &cache.ListWatch{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) { options.FieldSelector = fields.SelectorFromSet(fields.Set{"metadata.name": podName}).String() obj, err := c.CoreV1().Pods(podNamespace).List(context.TODO(), options) return runtime.Object(obj), err }, WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { options.FieldSelector = fields.SelectorFromSet(fields.Set{"metadata.name": podName}).String() return c.CoreV1().Pods(podNamespace).Watch(context.TODO(), options) }, }, &v1.Pod{}, 0, cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { p, ok := obj.(*v1.Pod) if ok { checkPodStatusFunc(p) } }, UpdateFunc: func(oldObj, newObj interface{}) { p, ok := newObj.(*v1.Pod) if ok { checkPodStatusFunc(p) } }, DeleteFunc: func(obj interface{}) { p, ok := obj.(*v1.Pod) if ok { checkPodStatusFunc(p) } }, }, ) return controller } </ins> // GetContainerStartedTime returns the time when the given container started and error if any func GetContainerStartedTime(p *v1.Pod, containerName string) (time.Time, error) { for _, status := range p.Status.ContainerStatuses {

pull-kubernetes-e2e-kind-ipv6 Kubernetes e2e suite.[It] [sig-node] Probing container should mark readiness on pods to false and disable liveness probes while pod is in progress of terminating See also, possibly related: Kubernetes e2e suite.[It] [sig-node] Probing container should mark readiness on pods to false while pod is in progress of terminating when a pod has a readiness probe Not sure. At least as of July 27, 2022, but probably earlier than that. No response ! /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign we should use listwatch to monitor pod status changes accurately, this interval 2s may lose pod change and cause the test failure
we did the same recently in another test, didn't we? I can't find the PR
sorry, i have not committed yet. do you plan to fix this test again? if so, I can unassign
he, no sorry for the confusion, I thought you were the one that fixed it, maybe it was ? There was one e2e that was flake and was solved using a listwatch instead of polling it every few seconds, I was trying to find the PR to show you , in case it can give you some idea ... please go ahead and fix it :)
this one?
yeah, so my memory was good then, I've identified the right author :smile:
I didn't understand what you said at first, but now I understand :smile:


Question: if pattern.SnapshotType == "" { e2eskipper.Skipf("Driver %q does not support snapshots - skipping", dInfo.Name) } <del> if pattern.VolMode == v1.PersistentVolumeBlock { // TODO: refactor prepareSnapshotDataSourceForProvisioning() below to use // utils.CheckWriteToPath / utils.CheckReadFromPath and remove // redundant InjectContent(). This will enable block volume tests. e2eskipper.Skipf("This test does not support block volumes -- skipping") } </del> // Create a volume testVolumeSizeRange := t.GetTestSuiteInfo().SupportedSizeRange

<!-- Feature requests are unlikely to make progress as an issue. Instead, please suggest enhancements by engaging with SIGs on slack and mailing lists. A proposal that works through the design along with the implications of the change can be opened as a KEP: --In block volume tests were skipped because is not using , there are more details about how these functions are used in This is a test that uses the , if tested using the it should have , in addition the test cluster should have the Things that are needed for a first time contributor a development kubernetes cluster, preferably on a public cloud instance although I'm not sure if kind can be used a CSI driver like PD CSI Driver, the storage class should be setup to use the xfs fstype the volume snapshot CRDs and controllers installed in the cluster (kubernetes-csi/external-snapshotter#usage) /sig storage /area testing /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Feature requests are unlikely to make progress as an issue. --/area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : #### What would you like to be : /area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey team! I'd like to further investigate on this issue if possible?

/assign
/area test hey sure, I've more info about how to test it, it requires these components: a development kubernetes cluster, preferably on a public cloud instance although I'm not sure if can be used a CSI driver like , the storage class should be setup to use the fstype the volume snapshot CRDs and controllers installed in the cluster ()
thanks a lot !! I might be a bit slow though, I'm pretty new to the community. I appreciate the documentation you shared, the assignation and how to replicate it, I'll test it out and if I have some doubts, I'll get back with you :D
Hey are you still working on this issue? I am interested in this
Yes, I'm still working on this, but would totally be open to doing pair-programming if you're open to it? I would benefit from you for solving the problem since I'm still pretty new to the codebase and I'm still understanding the environment.
Hey Are you still working on this, I would like to do pair-programming if possible.
Hi there definitely!! Ping me out in slack in the Kubernetes workspace with "Diego Goytia"


Question: }, }), output: `Name:             bar <ins> Labels:           <none> </ins> Namespace:        foo Address: Default backend:  APIGroup: example.com, Kind: foo, Name: bar

When command 'kubectl describe ing' Name: test-ing Namespace: practice Address: 10.10.01.10 Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found) TLS: test terminates Rules: Host Path Backends ---- ---- -------- / test-svc:80 (<none) Annotations: nginx Events: <noneResult has no information about labels. But Pod, Deployment, Service resources has label information like below. Name: test-room-test-mjsbz Namespace: test Priority: 0 Node: aks-test-test-test/22.22.222. Start Time: Fri, 09 Jul 2021 05:45:26 +0000 Labels: app=test pod-template-hash=test role=mediatest Why ingress has no information about labels? Actually when delete ingresss with labels is working correctly. Kubernetes version (use ): 1.18.14 Cloud provider or hardware configuration: AKS (Azure) OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig
/sig
/sig cli
/assign
/assign
/triage accepted /remove-kind bug /kind feature


Question: if oldPod.ResourceVersion == newPod.ResourceVersion { return } <del> if sched.skipPodUpdate(newPod) { </del> <ins> isAssumed, err := sched.SchedulerCache.IsAssumedPod(newPod) if err != nil { utilruntime.HandleError(fmt.Errorf("failed to check whether pod %s/%s is assumed: %v", newPod.Namespace, newPod.Name, err)) } if isAssumed { </ins> return } if err := sched.SchedulingQueue.Update(oldPod, newPod); err != nil { utilruntime.HandleError(fmt.Errorf("unable to update %T: %v", newObj, err)) }

Currently the scheduler may add an assumed pod back to the queue if the update didn't include specific fields (see ). There doesn't seem to be a reason why we can't simply ignore all updates of assumed pods. Assumed pods are waiting to be bound, if binding succeeded then the pod is scheduled. If binding fails, the pod's latest version will be put back into the unschedulable queue anyways. Performance: we save on unnecessary scheduling attempts. This is a followup on the discussion we had in /sig scheduling /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : #### What would you like to be : /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What we basically need to do is: 1) remove function 2) in the places where we call , we should simply call
/assign


Question: "--disable-admission-plugins=ServiceAccount,StorageObjectInUseProtection", // force enable all resources so we can check storage. "--runtime-config=api/all=true", <ins> // enable feature-gates that protect resources to check their storage, too. "--feature-gates=EphemeralContainers=true", </ins> }, etcdConfig) defer server.TearDownFn()

What happened: Adding a validating webhook with resources section of or breaks ability to add Ephemeral Containers as described in What you expected to happen: Debug command should attach to the running pod and allow user to interact with the ephemeral container image. This works when no validating webhook is configured. How to reproduce it (as minimally and precisely as possible): Using a K8 cluster with Ephemeral Containers feature gate enabled and v1.18.3 a validating webhook (contents don't matter except that resources is "pods/") <details</summary</detailsa test pod <details</summary</detailsto add an Ephemeral Container using alpha debug command Error: Anything else we need to know?: When using , can see that the api server appears to be throwing the error when trying to convert the object for the webhook invocation. API returning the 500 error: Environment: Kubernetes version (use ): v1.18.3 Cloud provider or hardware configuration: Minikube OS (e.g: ): MacOS Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
This looks like an issue with the ephemeral container storage. It is inheriting the New function from the pod storage to return a core.Pod{}, but that doesn't match the type the ephemeral storage actually expects. I think the ephemeral storage should probably set up its own New method to return the proper type. Enabling the feature gate for the ephemeral container feature in the webhook admission integration test would exercise this with webhooks as well. /sig node /priority important-longterm /assign
/remove-sig api-machinery
ack, thanks for the report & triage! I'm digging into it now.
I thank what matters for proper typing is that for returns the proper type. This is already implemented for and returns an API object. Here's what I think is happening: gets an update for the subresource. This is translated into an update of the pod spec and handed off to the pod store since ephemeral containers do not have separate storage. update logic triggers admission control for the object with the attributes of the subresource. webhook client attempts to convert the stored object (a ) to the Kind of the ephemeralcontainers attributes ( identified by ), but this conversion doesn't exist.

It is actually possible to convert from to , and this fixes the problem except that: The conversion is lossy, which would presumably be a problem for a mutating admission controller that returned an which could not be converted back into a This would result in the being sent in the instead of the , which is not the behavior we want. We could instead implement in , which allows us to override the Kind associated with . This results in no conversion and the is sent to the admission controller, but this also changes discovery info from: to: This fixes the problem, but I'm not sure what other side effects to expect. I imagine if we publish for this subresource then we should accept in addition to . I feel like I might be missing something in the storage configuration, though. do you know of any other possible fixes I could investigate?
For future reference, here's a trace from the validation dispatch that receives the conversion error:
mutating admission is applied here: validating admission methods (createValidation / updateValidation) is called from inside here: You need to wrap the createValidation / updateValidation methods (which expect an EphemeralContainers object) with methods that take a pod (which is what will pass in) and translate it to an EphemeralContainers object. See ScaleREST#Update methods for examples of this for the Scale subresource. Mutating admission is applied earlier, the only problematic call is validating admission Isn't it? The user is submitting an object... so that's what should be sent.
Thanks that set me straight. I have a working fix for this now, but I'm going to make sure it's covered in the e2e tests before sending it. re: for , it was a comment you made long ago that made me think there was a requirement to the contrary, so I must have misunderstood.
I had a chance to re-test this flow with which contains the fix to Although the original issue appears to be fixed, I am hoping to understand what the end goal is for ephemeral containers and their webhook behaviour. In my test, I enabled my to forward all resources . When running to add an ephemeral container to an existing pod, the only webhook that I received was for , which did contain the updated pod spec. However I was hoping that the addition of an ephemeral container would cause a webhook to trigger that wasn't of type . Maybe of type or ?

Is there any plans to add a webhook trigger for ephemeral containers specifically? API server logs after doing :
Hi Your admission controller should receive a review request for resource subresource with a kind . When I run a test on using the test webhook and the config: I see this request
I see it have had a misconfiguration on my previous test. Sorry about that. Thanks for your work on this! Having ephemeral container support is very useful!
Great! Thanks for trying out the feature while still in alpha and reporting the bug!


Question: // excludeCIDRs is a list of CIDR's which the ipvs proxier should not touch // when cleaning up ipvs services. ExcludeCIDRs []string `json:"excludeCIDRs"` <ins> // strict ARP configure arp_ignore and arp_announce to avoid answering ARP queries // from kube-ipvs0 interface StrictARP bool `json:"strictARP"` </ins> } // KubeProxyConntrackConfiguration contains conntrack settings for

/kind bug Kubernetes version: 1.13.1. kube-proxy mode: IPVS. SDN: flannel. We are used services like this: where IP 192.168.10.201 is from our internal /24 network (in particular, our Kube nodes have IPs 192.168.10.1, 192.168.10.2 etc). Note that there is no conflicts between real IPs and virtual external ones. Before updating from 1.12.3 to 1.13.1, it worked correctly. In partucular these IPs was accessible from other non-Kubernetes hosts from 192.168.10.0/24 network. After update kube-proxy to 1.13.1, this feature stops working: non-Kubernetes hosts cannot resolve service IPs via ARP: On Kubernetes hosts it still working, because not requires ARP resolving. If we reset sysctl parameter arpignore on Kube hosts, all works normally even from non-Kube hosts. Quck debugging shows that problem is in the commit . Please make this behaviour opt-out. (Now we resets arpignore to 0 via cron every 1 min.)
/sig network
Having the same problem on Kubernetes 1.13.2 btw Thanks for a workaround!
Same problem, fix this please!!!
This is similar to We've had a discussion there on how to best implement a solution but haven't decided yet I'm in favor of flag which defaults to the previous behavior to avoid breaking existing setups (alternatives are cleaner but involve creating an additional arptable rule per service which I think we should avoid) what do you think?
OK

What happened: After PR: , we noticed communications issue in our pods. We use the Lyft CNI plugin () which relies on two interfaces in pod: an IPVLAN interface connecting to the AWS VPC which has a VPC IP an Unnumbered point-to-point interface (veth) connecting to the host and that does not have an IP (it "borrows" the one from the IPVLAN interface) On the host side, there is a veth interface which also doesn't have an IP (it borrows one from the main interface). With the latest versions of kube-proxy communication between host and pods break What you expected to happen: Communication still works How to reproduce it (as minimally and precisely as possible): Use the Lyft CNI plugin Deploy kube-proxy Try to ping the host from a pod (or a pod from the host) Anything else we need to know?: The new sysctls create 2 issues: containers started after kube-proxy inherit so they will not answer ARP queries on the veth interface the veth on the host side also does not have an IP so it will no answer ARP queries from the pods I know that our setup is pretty uncommon but assuming that the veth on the host side will have an IP seems dangerous. Some discussions on the PTP plugin for instance suggest not using IP for the host interface: containernetworking/plugins Possible fixes: create a flag to enable ARP filtering (I think to avoid breaking existing setups, we should disable it by default but I'm fine with enabling it by default as long as we document the solution) use arptables to filter ARP queries for clusterIPs (): a bit more work because we need to do it for all clusterIPs (there may be a more efficient way to do it but I could not make it work by filtering on the kube-ipvs interface) and this requires the arptables binary. We could test for the existence of the binary and log a warning if it is missing (similar to behavior with conntrack) Environment: Kubernetes version (use ): kube-proxy OS (e.g. from /etc/os-release): Kernel (e.g. ): /area ipvs /sig network /kind bug /assign ping
I can say only that solution with arptables working fine with MetalLB.
Oh, really? Glad you confirm that.

But I would like to make sure that we won't create too many arptables rules. Can ipset works with arptables?
It doesn't look like arptables works with ipset
Unfortunately doesn't support ipsets, but new do. (not fully implemented yet) Eg we can create ipset for arp: But current implementation is not allowing to use it for arp rules for now: Will return: I guess because there is still no type specified for (destination ip) and (source ip) BTW: nft does not allow to create set of integer components: But there is a way for specify multiple IPs in one rule, eg: Where is IPv4 addresses in decimal format More info here:
Another option would be to use the and filter all queries but this flag is not in kube-proxy today
Bad idea, it will handle but not - which can be really differ
can you explain more detail why Lyft CNI can not assign IPs to the right interfaces for avoid this kind of issues?
Yes you are right. bad idea It could definitely add IPs to these interfaces but it's not the choice that was made. More details here: ("Unnumbered point-to-point interface"). I could probably PR a fix there. I'm a bit worried that some other CNI plugins could break here though
We are about to deploy some 1.13 clusters and can't use kube-proxy 1.13 as is We could start with a custom build but I assume others will be impacted The simplest way forward for now is probably to use a flag to control this behavior. I think the safest is probably to disable this by default to avoid the issue for people upgrading but if you think it is really important we can enable it by default. I know we'd like to avoid adding more flags but other solutions seem a bit more complicated and will most likely result in one iptables rule per service. What do you think ?
Just FYI, we have another CNI plugin that is affected by this.
We addressed the issue by assigning a link-local address 169.254.1.1/32 to the host-side veth in . So this issue is not critical at least for our case.
Good news everyone, new nft supports setting ip address for arp filters : that's makes possible for using sets


Question: HostConfig: hc, } <del> // Apply platform-specific options. if err := ds.applySandboxPlatformOptions(hc, c, createConfig, image, securityOptSeparator); err != nil { </del> <ins> // Apply linux-specific options. if err := ds.applySandboxLinuxOptions(hc, c.GetLinux(), createConfig, image, securityOptSeparator); err != nil { </ins> return nil, err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: is trying to setup DNS for Windows containers, so that Pods on Windows could confirm to dns policies. However, as posted Docker is not managing the networking here for windows. So it doens't really care about any network settings. So passing it to docker shim's hostconfig also doens;t make sense here. DNS for Windows containers will be set by CNI plugins. So PR reverts . This issue tracks the process of pushing DNS configure into CNI and passed that from Kubelet to container runtime. What you expected to happen: Pods on Windows confirm to dns policies. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig windows
Need a linked issue & PR in CNI repo too so that capability arguments can be validated from kubelet
Fixed in by dns capabilities, Azure CNI has already supported this. /close
Closing this issue. <detailsIn response to : Fixed in by dns capabilities, Azure CNI has already supported this. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func printEvent(event *api.Event, w io.Writer) error { _, err := fmt.Fprintf( <del> w, "%st%st%st%st%st%sn", </del> <ins> w, "%st%st%st%st%st%st%sn", </ins> event.Timestamp.Time.Format(time.RFC1123Z), event.InvolvedObject.Name, event.InvolvedObject.Kind, <ins> event.InvolvedObject.FieldPath, </ins> event.Condition, event.Reason, event.Message,

Forked from ... The event generated from kubelet for BoundPod and displayed by kubectl should have Container(s) column ; otherwise, the user might think the entire pod is at the running state, but actually only one or some containers are running. When there are missing event(s) for failed containers, the thing gets worse. For example, in case, only net container is running. $ get events | grep testpd Running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get events Mon, 29 Dec 2014 20:58:38 +0000 testpd BoundPod waiting created Created with docker id eb3f53865a8825a3db3b64f6f319b64bfecbcbabc7115a73c2358c8c73793004 Mon, 29 Dec 2014 20:58:38 +0000 testpd Pod Pending scheduled Successfully assigned testpd to kubernetes-minion- Mon, 29 Dec 2014 20:58:38 +0000 testpd BoundPod waiting pulled Successfully pulled image kubernetes/pause:latest Mon, 29 Dec 2014 20:58:39 +0000 testpd BoundPod running started Started with docker id eb3f53865a8825a3db3b64f6f319b64bfecbcbabc7115a73c2358c8c73793004


Question: <tspan x="142.476" y="866.282"></tspan> </text> <g> <del> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 921.724,322.317 A 70.7456,70.7456 0 0 0 790.8,322"/> <polygon style="fill: #000000" points="924.195,329.333 916.879,320.879 922.342,322.066 926.569,318.408 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="924.195,329.333 916.879,320.879 922.342,322.066 926.569,318.408 "/> </del> <ins> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="758" y1="273" x2="782.332" y2="408.717"/> <polygon style="fill: #000000" points="783.655,416.099 776.969,407.138 782.332,408.717 786.812,405.374 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="783.655,416.099 776.969,407.138 782.332,408.717 786.812,405.374 "/> </ins> </g> <g> <del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="790.8" y1="331.5" x2="811.94" y2="331.5"/> <polygon style="fill: #000000" points="819.44,331.5 809.44,336.5 811.94,331.5 809.44,326.5 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="819.44,331.5 809.44,336.5 811.94,331.5 809.44,326.5 "/> </g> <g> <rect style="fill: #ffffff" x="876.576" y="75.6768" width="70.2" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="876.576" y="75.6768" width="70.2" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="911.676" y="98.5768"> <tspan x="911.676" y="98.5768">Firewall</tspan> </del> <ins> <rect style="fill: #ffffff" x="941.576" y="75.6768" width="70.2" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="941.576" y="75.6768" width="70.2" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="976.676" y="98.5768"> <tspan x="976.676" y="98.5768">Firewall</tspan> </ins> </text> </g> <g> <del> <path style="fill: #ffffff" d="M 886.242 -47.953 C 876.87,-48.2618 858.694,-41.7773 861.25,-27.8819 C 863.806,-13.9865 876.018,-10.8988 881.13,-14.9129 C 886.242,-18.9271 873.178,4.54051 898.17,10.7162 C 923.161,16.8919 935.941,7.01079 932.249,-0.0912821 C 928.557,-7.19336 954.117,16.5832 966.045,2.99658 C 977.973,-10.59 953.833,-23.5589 958.945,-21.7062 C 964.057,-19.8535 979.677,-22.3237 974.565,-45.4827 C 969.453,-68.6416 923.445,-50.7321 928.557,-54.1287 C 933.669,-57.5253 920.889,-74.5086 904.986,-71.112 C 889.082,-67.7153 887.954,-61.5516 886.25,-47.965 L 886.242,-47.953z"/> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 886.242 -47.953 C 876.87,-48.2618 858.694,-41.7773 861.25,-27.8819 C 863.806,-13.9865 876.018,-10.8988 881.13,-14.9129 C 886.242,-18.9271 873.178,4.54051 898.17,10.7162 C 923.161,16.8919 935.941,7.01079 932.249,-0.0912821 C 928.557,-7.19336 954.117,16.5832 966.045,2.99658 C 977.973,-10.59 953.833,-23.5589 958.945,-21.7062 C 964.057,-19.8535 979.677,-22.3237 974.565,-45.4827 C 969.453,-68.6416 923.445,-50.7321 928.557,-54.1287 C 933.669,-57.5253 920.889,-74.5086 904.986,-71.112 C 889.082,-67.7153 887.954,-61.5516 886.25,-47.965 L 886.242,-47.953"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="922.428" y="-23.1971"> <tspan x="922.428" y="-23.1971">Internet</tspan> </del> <ins> <path style="fill: #ffffff" d="M 948.242 -47.953 C 938.87,-48.2618 920.694,-41.7773 923.25,-27.8819 C 925.806,-13.9865 938.018,-10.8988 943.13,-14.9129 C 948.242,-18.9271 935.178,4.54051 960.17,10.7162 C 985.161,16.8919 997.941,7.01079 994.249,-0.0912821 C 990.557,-7.19336 1016.12,16.5832 1028.04,2.99658 C 1039.97,-10.59 1015.83,-23.5589 1020.94,-21.7062 C 1026.06,-19.8535 1041.68,-22.3237 1036.56,-45.4827 C 1031.45,-68.6416 985.445,-50.7321 990.557,-54.1287 C 995.669,-57.5253 982.889,-74.5086 966.986,-71.112 C 951.082,-67.7153 949.954,-61.5516 948.25,-47.965 L 948.242,-47.953z"/> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 948.242 -47.953 C 938.87,-48.2618 920.694,-41.7773 923.25,-27.8819 C 925.806,-13.9865 938.018,-10.8988 943.13,-14.9129 C 948.242,-18.9271 935.178,4.54051 960.17,10.7162 C 985.161,16.8919 997.941,7.01079 994.249,-0.0912821 C 990.557,-7.19336 1016.12,16.5832 1028.04,2.99658 C 1039.97,-10.59 1015.83,-23.5589 1020.94,-21.7062 C 1026.06,-19.8535 1041.68,-22.3237 1036.56,-45.4827 C 1031.45,-68.6416 985.445,-50.7321 990.557,-54.1287 C 995.669,-57.5253 982.889,-74.5086 966.986,-71.112 C 951.082,-67.7153 949.954,-61.5516 948.25,-47.965 L 948.242,-47.953"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="984.428" y="-23.1971"> <tspan x="984.428" y="-23.1971">Internet</tspan> </ins> </text> </g> <g> <del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="912.986" y1="12.703" x2="911.878" y2="65.9428"/> <polygon style="fill: #000000" points="911.723,73.4412 906.932,63.3394 911.878,65.9428 916.929,63.5474 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="911.723,73.4412 906.932,63.3394 911.878,65.9428 916.929,63.5474 "/> </del> <ins> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="974.985" y1="12.703" x2="976.415" y2="65.9442"/> <polygon style="fill: #000000" points="976.616,73.4415 971.349,63.5793 976.415,65.9442 981.346,63.3109 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="976.616,73.4415 971.349,63.5793 976.415,65.9442 981.346,63.3109 "/> </ins> </g> <g> <del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="911.676" y1="113.677" x2="951.207" y2="293.491"/> <polygon style="fill: #000000" points="952.818,300.816 945.787,292.123 951.207,293.491 955.554,289.976 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="952.818,300.816 945.787,292.123 951.207,293.491 955.554,289.976 "/> </del> <ins> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="976.676" y1="113.677" x2="978.16" y2="218.265"/> <polygon style="fill: #000000" points="978.266,225.764 973.125,215.836 978.16,218.265 983.124,215.694 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="978.266,225.764 973.125,215.836 978.16,218.265 983.124,215.694 "/> </ins> </g> <g> <del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="939.022" y1="341" x2="803.77" y2="366.309"/> <polygon style="fill: #000000" points="796.398,367.689 805.308,360.935 803.77,366.309 807.147,370.764 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="796.398,367.689 805.308,360.935 803.77,366.309 807.147,370.764 "/> </del> <ins> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="978.298" y1="266" x2="977.033" y2="358.365"/> <polygon style="fill: #000000" points="976.931,365.864 972.068,355.797 977.033,358.365 982.067,355.934 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="976.931,365.864 972.068,355.797 977.033,358.365 982.067,355.934 "/> </ins> </g> <g> <del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="953.298" y1="341" x2="970.506" y2="360.758"/> <polygon style="fill: #000000" points="975.431,366.414 965.093,362.157 970.506,360.758 972.634,355.589 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="975.431,366.414 965.093,362.157 970.506,360.758 972.634,355.589 "/> </g> <g> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="967.572" y1="341" x2="1149.96" y2="366.739"/> <polygon style="fill: #000000" points="1157.39,367.788 1146.79,371.341 1149.96,366.739 1148.18,361.439 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="1157.39,367.788 1146.79,371.341 1149.96,366.739 1148.18,361.439 "/> </del> <ins> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="992.573" y1="266" x2="1174.02" y2="363.492"/> <polygon style="fill: #000000" points="1180.63,367.042 1169.45,366.713 1174.02,363.492 1174.19,357.904 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="1180.63,367.042 1169.45,366.713 1174.02,363.492 1174.19,357.904 "/> </ins> </g> <g> <rect style="fill: #ffffff" x="-54" y="370.5" width="562" height="383.25"/>

CA Advisor runs inside docker containers, but I don't believe it's correct to show the Kubelet and Proxy processes as inside the docker box on the minion. Also, shouldn't the internet-firewall-really load balance across minion proxies instead of be connected to just one SPOF minion, treating that connection more like a logical connection across all minion proxies?
This is the desired state, which is admittedly not the current state. Yeah, this is a good point. k8s currently relies on its environment for load balancing. Maybe or can speak to the desired state here.
We should just review the diagram and make it correctly reflect the state of 1.0.


Question: }, { "ImportPath": "github.com/spf13/cobra", <del> "Rev": "6644d46b81fa1831979c4cded0106e774e0ef0ab" </del> <ins> "Rev": "c439c4fa093711d42e1b01acb1235b52004753c1" </ins> }, { "ImportPath": "github.com/spf13/pflag",

BUG REPORT: /kind bug What happened: Arguments to a command inside pod are treated as kubectl flags and deprecation warnings are shown for them from . What you expected to happen: Arguments to a command inside pod should not be treated as kubectl flags by . How to reproduce it: The arguments , , , passed to the command inside the pod above are treated as flags and deprecation warnings are shown for them from . Environment: Kubernetes version: 9: Cloud provider or hardware configuration: Ubuntu Xenial VM. OS:Kernel:Install tools: minikube:
/sig cli
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig cli Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is it ok if we remove this warning ?
the warning should be removed just for the arguments to a command inside pod that are passed after the , and not for the arguments to itself. The warning is still valid and needed for arguments with underscore () to itself.
This is a bug of cobra. I've sent out a PR to fix it. And bumped to the fixed version in .
Thanks for fixing this!


Question: } // Check cache for existing provider. <del> if provider, ok := cache.getClient(issuer, clientID); ok { </del> <ins> if provider, ok := cache.getClient(clusterAddress, issuer, clientID); ok { </ins> return provider, nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We using go-code kubectl cli package. We are a long running app serving to many different Kubernetes clusters. Each cluster has a different user credentials and bearer token to access it. But they all use the same issuer and clientid. The oidc cache is keyed off of these two only. So what happens is when we switch to another configuration it pulls in the oidc provider for the other cluster and that bearer token does not have authority to this other cluster. The cluster config has the correct info in it but the oidc provider ignores that and used the stale data. What you expected to happen: oidc provider would always use the appropriate current config context to access the cluster How to reproduce it (as minimally and precisely as possible): In same app instance create two cluster configs to two different clusters with same oidc issuer and clientid by DIFFERENT credentials. Where the credentials on one are not authorized for the other cluster. Try to get namespaces on each, one will work and the other won't. Anything else we need to know?: Environment: Kubernetes version (use ): latest go-code from as a package called from go application.Cloud provider or hardware configuration: IBM Kubernetes OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: package devicemanager import ( <ins> "encoding/json" </ins> "testing" "github.com/stretchr/testify/require" <ins> "k8s.io/apimachinery/pkg/util/sets" pluginapi "k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1" </ins> "k8s.io/kubernetes/pkg/kubelet/cm/devicemanager/checkpoint" )

When calling GetAllocatableDevices, the API returns all the registered devices, regardless of their health status. Thing is, the - and in general doesn't make much sense to report as allocatable Unhealthy devices, as is also documented in the (see last paragraph) The API should not report unhealthy device. This means the reported device set can potentially change across multiple invocations of the API, as . a kube = 1.21 environment, enable the feature gate any device plugin - including sample device plugin sure 1+ devices are reported unhealthy the podresources GetAllocatableResources API, observe the unhealthy device is reported in the return value. N/A Kubernetes version (use ): 1.21 and beyond Cloud provider or hardware configuration: not relevant OS (e.g: ): any Kernel (e.g. ): not relevant Install tools: not relevant Network plugin and version (if this is a network-related bug): not relevant Others: not relevant
/sig node
/triage accepted