Question: }{ { api.RestartPolicyAlways, <del> []string{"list", "list", "list", "inspect_container", "inspect_container", "inspect_container", "create", "start", "create", "start", "list", "inspect_container", "inspect_container", "inspect_container"}, </del> <ins> []string{"list", "list", "list", "inspect_container", "inspect_container", "inspect_container", "create", "start", "create", "start", "list", "inspect_container", "inspect_container", "inspect_container", "inspect_container", "inspect_container"}, </ins> []string{"succeeded", "failed"}, []string{}, }, { api.RestartPolicyOnFailure, <del> []string{"list", "list", "list", "inspect_container", "inspect_container", "inspect_container", "create", "start", "list", "inspect_container", "inspect_container", "inspect_container"}, </del> <ins> []string{"list", "list", "list", "inspect_container", "inspect_container", "inspect_container", "create", "start", "list", "inspect_container", "inspect_container", "inspect_container", "inspect_container"}, </ins> []string{"failed"}, []string{}, },

Currently kubelet reports current status only. If pod's restart policy is not never, the last terminated container status will be override by the latest status, and lost. We should report last terminated container status. I am going to update our v1beta3 api first, but will hold the implementation until we have
did you have a chance to start working on it? IIUC, to support this we need to change the api (), which IIUC from what wrote in some other thread should be done this week. I'm happy to help with it if you want me to help.
Only breaking changes have a deadline. Additive changes can be done at any time.
I am working on this. PR should be ready today or tomorrow.
Also agree with This is not part of API breakage.
Thanks for answers & clarifications and


Question: c.registeredPods[key] = pod if prev != nil { for name := range getConfigMapNames(prev) { <ins> // On an update, the .Add() call above will have re-incremented the // ref count of any existing items, so any configmaps that are in both // names and prev need to have their ref counts decremented. Any that // are only in prev need to be completely removed. This unconditional // call takes care of both cases. </ins> c.configMapStore.Delete(prev.Namespace, name) } }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Occasionally (more frequently if the node is heavily loaded) the node fails to mount a secret or configmap for a pod. Error recovery fails and the pod must be replaced with a new pod to have any hope of getting the secret or configmap volume mounted. The problem appears to be that if a pod is updated, its secrets are removed from the secretStore's secrets map. See: Normally the effects of the bug are not felt since the usual order of operations masks the bug: 1) The pod is created and registered the first time with the secrets manager 2) The volume mounter gets the secret 3) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) If the race is lost, though, then the bug symptoms are manifest: 1) The pod is created and registered the first time with the secrets manager 2) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) 3) The volume mounter tries to get the secret and fails with the message What you expected to happen: On update, the secrets manager should only delete secrets if they are not present in the latest pod definition (rather than deleting all prior secrets). Whether the pod update happens before or after the volume mounter attempts to get the secret, it should succeed. How to reproduce it (as minimally and precisely as possible): Because it is a race condition, it's difficult to reproduce. We've most often seen the bug on systems that were under load. 1) Create a pod that has a secrets or configmap volume 2) See if the pod fails to mount the volume. Anything else we need to know?: If the above analysis is correct, this bug has been present since 1.6 for secret volumes, and 1.7 for configmap volumes. Environment: We've seen the bug in software based upon the 1.6 release.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
The PR addresses this issue. /sig node
This may be a false alarm. I've never actually reproduced it in Kubernetes proper, only in OpenShift. Since our smoking gun was anything but, I'm going to close this until I manage to find an actual problem in Kubernetes.
To close the loop on this for any that find it via future searches: It turns out that this was a false alarm. The above analysis was flawed and this turns out not to be a bug (or maybe a very minor one if we consider it a bug to log a an error on a (semantic) non-error). The error messages like were not an indicator of a failure to mount a secret volume, but instead a failure to update a secret volume during pod teardown. Here's how the error message is triggered: pod is created volume manager gets the secret and mounts the volume the lifetime of the pod, the volume manager periodically re-checks the secret to make sure that it's still in a good state. pod goes away volume is unmounted If one of the refreshes from step 3 happens after step 4 but before step 5, the log message will appear.


Question: func (os *OpenStack) LoadBalancer() (cloudprovider.LoadBalancer, bool) { glog.V(4).Info("openstack.LoadBalancer() called") <ins> if reflect.DeepEqual(os.lbOpts, LoadBalancerOpts{}) { glog.V(4).Info("LoadBalancer section is empty/not defined in cloud-config") return nil, false } </ins> network, err := os.NewNetworkV2() if err != nil { return nil, false

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I use but my cloud provider doesn't support Openstack LBaaS. Because of the latter I've been using MetalLB instead to get things going. However I keep getting warnings when deploying a when there is no configuration defined in . NOTE: has only a section - there is no mention no section. What you expected to happen: should not print warning via Openstack LoadBalancer since it doesn't support/can't do anything with it. How to reproduce it (as minimally and precisely as possible): Configure Push a Create a Anything else we need to know?: See this Environment: See this
/sig cloud-provider
/sig openstack
that error is coming from row but I am not 100% sure should we support metallb in openstack cloud provider integration. Metallb is not part of openstack. any opinion? anyways it is little bit weird that EnsureLoadBalancer function is even working if LoadBalancer section is not defined in config. Openstack lbaas is not initialized but that function is called?
if there is no LBaaS in the environment and or no , we should not try to deploy one (and let metallb to do the work as it is supposed to). i think.
when you start node, do you see this log print in logs? Can you try to add following to does creating service with type loadbalancer (with metallb) work after that without error prints?
~ With the original config doesnt show up in the log files. The only reference for in the logs is:~ UPDATE BELOW Now, if adding as it yields: And: The IP allocation via MetalLB works The service is remotely available The warning on object disappears:
weird, without this Loadbalancer thing in config it should end up to row and print "Claiming to support LoadBalancer". Now when you that Loadbalancer section with random lb-version it ends up to row and it does not have support to external loadbalancer from openstack. When using metallb the init function in openstack should return false (no support from openstack) but I do not have now idea why it does not do that (when you do not have that loadbalancer section in config)
I take it back. my log level was too high ().

After moving to it shows up when there is no section in :
cool, then we should have check in function as first thing which checks is LoadBalancer even defined in config - if no then return nil, false
(or someone else openstack guy) do you have time to run some test to fix this? I am on vacation and I do not have my test environment with me :) I am thinking to add following to row So I am trying to check is lbOpts totally empty, if yes - then it does not initialize openstack lbaas at all. Maybe I will do PR, but I do not have possibility to test it


Question: Requests: podRes, Limits:   podRes, }, <del> }, framework.PodStartShortTimeout) </del> <ins> }) </ins> defer func() { // Clean-up the critical pod

master-blocking kind-ipv6-master-parallel RuntimeClass should run a Pod requesting a RuntimeClass with scheduling without taints AdmissionWebhook [Privileged:ClusterAdmin] should honor timeout [Conformance] 01-27-2023 16:08 PST https://k8s- { failed [FAILED] timed out while waiting for pod runtimeclass-3517/without-label to be running In [It] at: @ 01/28/23 22:33:45.607 } No response kind cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It looks like that “Kubernetes e2e suite: . Would we want to reopen this issue? I will keep a close eye on to see if it continues to flake.
Reopen to monitor failures cc
/sig node /cc
/retitle [It] [sig-node] RuntimeClass should run a Pod requesting a RuntimeClass with scheduling without taint
Example failure:
Failure The pod was created at , from the kubelet logs: so the pod started at , but it was deleted at because the 1 minute timeout was hit. It looks like the container startup took a while based on containerd logs: there is a gap there of and between and
It is hitting this timeout - That uses which is 1 min which seems to be too short. We should probably use or which is 5min or 2min, because the whole pod needs to start and it seems like 1min is too short in this case.


Question: name := "/" + c.Name id := GetFakeContainerID(name) f.appendContainerTrace("Created", id) <ins> timestamp := f.Clock.Now() </ins> // The newest container should be in front, because we assume so in GetPodStatus() f.RunningContainerList = append([]dockertypes.Container{ <del> {ID: id, Names: []string{name}, Image: c.Config.Image, Labels: c.Config.Labels}, </del> <ins> {ID: id, Names: []string{name}, Image: c.Config.Image, Created: timestamp.Unix(), State: statusCreatedPrefix, Labels: c.Config.Labels}, </ins> }, f.RunningContainerList...) f.ContainerMap[id] = convertFakeContainer(&FakeContainer{ <del> ID: id, Name: name, Config: c.Config, HostConfig: c.HostConfig, CreatedAt: f.Clock.Now()}) </del> <ins> ID: id, Name: name, Config: c.Config, HostConfig: c.HostConfig, CreatedAt: timestamp}) </ins> f.normalSleep(100, 25, 25)

https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory.

Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.


Question: if !util.AccessModesContainedInAll(v.plugin.GetAccessModes(), v.options.PVC.Spec.AccessModes) { return nil, fmt.Errorf("invalid AccessModes %v: only AccessModes %v are supported", v.options.PVC.Spec.AccessModes, v.plugin.GetAccessModes()) } <del> if util.CheckPersistentVolumeClaimModeBlock(v.options.PVC) { return nil, fmt.Errorf("%s does not support block volume provisioning", v.plugin.GetPluginName()) } </del> volSpec, err := v.manager.CreateVolume(v) if err != nil {

FEATURE REQUEST: vSphere volume plugin does not support volume device yet. Need to add volumeMode in PV spec as other plugin did. release 1.12
/sig storage


Question: return ret } <ins> // a simple token source simply always returns the token property </ins> type fakeTokenSource struct { <del> expiresOn   string accessToken string </del> <ins> token        *azureToken tokenCalls   uint tokenErr     error refreshToken *azureToken refreshCalls uint refreshErr   error </ins> } func (ts *fakeTokenSource) Token() (*azureToken, error) { <del> return &azureToken{ token:       newFackeAzureToken(ts.accessToken, ts.expiresOn), environment: "testenv", clientID:    "fake", tenantID:    "fake", apiserverID: "fake", }, nil </del> <ins> ts.tokenCalls++ return ts.token, ts.tokenErr } func (ts *fakeTokenSource) Refresh(*azureToken) (*azureToken, error) { ts.refreshCalls++ return ts.refreshToken, ts.refreshErr </ins> } func token2Cfg(token *azureToken) map[string]string {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, when AAD login is configured, refresh token timeout would make kubectl print an error, not able to re-login without manually remove the tokens. Expect kubectl to promote user to login again. Why is this needed: Configure refresh-token timeout from Azure is a way to force re-login after inactivity. kubectl doesn't promote re-login makes users harder to apply timeout policy.
/sig azure
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig platform/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure
the root cause of this issue is in this first gets the cached token (result of device code flow) at it's trying to refresh an expired token at and failed. An error is returned. Since the expired token is still in the cache, it will always hit this path until user removes the token from kubeconfig. the fix should be: the token from cache a retry to re-initiate the device code flow in
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig platform/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
I'm working on it.
are you working on it? I will grab it if I dont hear it from you :)
have the code ready will try PR this weekend Best, Ping
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Closing as merged


Question: reconciler.volumesNeedUpdateFromNodeStatus = append(reconciler.volumesNeedUpdateFromNodeStatus, volumeName1, volumeName2) // Act - run reconcile loop just once. // "volumesNeedUpdateFromNodeStatus" is not empty, so no unmount will be triggered. <del> reconciler.reconcileNew() </del> <ins> reconciler.reconcile() </ins> // Assert assert.True(t, reconciler.StatesHasBeenSynced())

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We have even encountered such case that pod continuously reports “Unable to attach or mount volumes for pod;...” when kubelet restarted. The reasons are listed in details as below: , as result of apiserver instantaneous unstablility, dswp get pvc‘s pv spec failed, then the pvc volume does not be to dsw. , syncStates() lists all the volume directories locally, and recontructs the volume from local volume, if the volume isn't in dsw, it will put the recontructed volume to asw, but now the volume.OuterVolumeSpecName isn't correct. chance, at this time, In , the fail to pvc volume is to dsw again normally in reconcile() -mountAttachVolumes(), it compares dsw and asw by volume's innerVolumename, it's all the same. but it could not sense the OuterVolumeSpecName dismatch. so it will not re-mount again to update asw cache , it checks pod's volume attach or mount success by comparing volume.OuterVolumeSpecName and , but this pv'c OuterVolumeSpecName isn't correct in asw. so pod continuously reports “Unable to attach or mount volumes for pod;...” reconcile() could sense the reconstruct volume's dismatch to the dsw, then re-mount again to make volume cache consistency Kubernetes version (use ): 1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
CC
I think this is a point that can be optimized. /cc
I will give a fix for this
Thanks for reporting the issue, this innerSpecName and outerSpecName are confusing and cause issues. Please ping me for review when you have a fix. Thanks!
Please don't triage your own issues immediately, give someone else a chance to review them :( /sig storage
okay, I know


Question: if stdout != nil { go io.Copy(stdout, p) } <del> return command.Wait() </del> <ins> return newRktExitError(command.Wait()) </ins> } if stdin != nil { // Use an os.Pipe here as it returns true *os.File objects.

With rkt as the container runtime, the e2e test fails with I dove into it a bit further and it looks like a real issue. With additional logging it appears that the probe does fail (exec happens and returns incorrectly as expected), but when calls it never gets that the given ID exists. The pod definition I've been using is: It restarts very slowly with rkt (on the order of an hour I believe, see sleep 600), but restarts very frequently with docker. cc


Question: }, expected: &value1, }, <ins> { desc: "should return correct value if elapsed UsageCoreNanoSeconds exceeds 18446744073", stats: &runtimeapi.ContainerStats{ Attributes: &runtimeapi.ContainerAttributes{ Id: "1", }, Cpu: &runtimeapi.CpuUsage{ Timestamp: int64(time.Second / time.Nanosecond), UsageCoreNanoSeconds: &runtimeapi.UInt64Value{ Value: 68172016162105, }, }, }, cpuUsageCache: map[string]*cpuUsageRecord{ "1": { stats: &runtimeapi.CpuUsage{ Timestamp: 0, UsageCoreNanoSeconds: &runtimeapi.UInt64Value{ Value: 67983588375722, }, }, }, }, expected: &value2, }, </ins> } for _, test := range tests {

What happened: CRI CPU metrics are incorrect when using a large number of cores, due to a overflow. What you expected to happen: CPU metrics should be correct for arbitrarily high numbers of cores. How to reproduce it (as minimally and precisely as possible): Deploy a pod with 20 CPU units allocated, and fetch the CRI metrics. The metrics will be incorrect. Environment: Kubernetes version (use ): 1.14.4 CRI: CRI-O cc cc /sig node /sig scalability
How did you get the cri cpu metrics?
kubelet /stats/summary API. For "large" pods, we confirmed that cri-o/cgroupfs calculations were being generated correctly and CPU data overflows at the kubelet CRI stats provider layer. Overflows of CPU data in /stats/summary results in metrics-server having bad data which further propagates to the HPA scaling Pods incorrectly.


Question: Command:       cmd, Namespace:     t.f.Namespace.Name, PodName:       t.utilPod.Name, <del> ContainerName: "util", </del> <ins> ContainerName: t.utilPod.Spec.Containers[0].Name, </ins> CaptureStdout: true, CaptureStderr: true, })

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-netwok jobs running serial tests Which test(s) are failing: [sig-network] DNS configMap nameserver.* Since when has it been failing: Seems this was the trigger Testgrid link: https://k8s-,%20master Reason for failure: It seems is trying to use the container Anything else we need to know:
/sig network /sig testing /cc /triage accepted
/assign


Question: err error, ) <ins> const ( initialDuration = 1 * time.Second factor          = 2.0 steps           = 5 ) </ins> // newV1NodeClient creates a new NodeClient with the internally used gRPC // connection set up. It also returns a closer which must to be called to close // the gRPC connection when the NodeClient is not used anymore.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: If kubelet plugin registration fails during the registration process, the kubelet never attempts to register the socket again (assuming the error was transient), unless the socket for the driver is deleted & recreated to trigger a new registration. What you expected to happen: Maybe retry with exponential backoff when kubelet plugin registration fails instead of just logging and waiting for another filesystem event like it does today. Of course, if a socket returns a terminal error (does not implement the kubelet plugin registration service, for example), we should not retry. Follow up on /milestone v1.14 /sig storage How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'd like to work on this issue
/assign
Re opening. was a temporary work around for this. is the permanent fix.
Hello! We are starting Bug Freeze for 1.15. Is this issue still planned on being incorporated into 1.15?
the dependent PR is not yet merged. can we get this in for 1.15?
said that is the "permanent fix." and that has merged. So let's close this out /close
Closing this issue. <detailsIn response to : said that is the "permanent fix." and that has merged. So let's close this out Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: etcdObjectCounts = compbasemetrics.NewGaugeVec( &compbasemetrics.GaugeOpts{ Name:              "etcd_object_counts", <del> DeprecatedVersion: "1.21.0", </del> <ins> DeprecatedVersion: "1.22.0", </ins> Help:              "Number of stored objects at the time of last check split by kind. This metric is replaced by apiserver_storage_object_counts.", StabilityLevel:    compbasemetrics.ALPHA, },

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --https://k8s- 03-27 Two different failure reasons /assign /milestone 1.20 /priority critical-urgent /cc
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , ] Use to clear the milestone. <detailsIn response to : <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --https://k8s- #### Which test(s) are failing: 03-27 #### Testgrid link: Two different failure reasons #### Anything else we need to know: /milestone 1.20 /priority critical-urgent /cc </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.21
/sig scalability cc
In both cases we have no master logs which makes it nearly impossible to debug this. should fix the issue with missing logs. In the meantime, I'm running a manual run of the same config as the last failing run
one of the reasons for failure is: We also see quite a few OOMs on ~10 nodes. The processes that are oom killed looks quite random: pause pods, system-logind, systemd-journal. It's quite unlikely that those are the processes that eats all the memory
I checked serial console logs for one of nodes where OOM has been reported and most likely metrics-server is a component that consumes a lot of memory and triggers OOM killer on nodes. In my manual run that is currently happening we see that it fails and jumps from node to node triggering OOMs on another node:
From successful run: There are multiple instances of metrics-server, but all of them in a short period of time (when the cluster starts). They are recreated by podnanny. It looks like in failed runs podnanny for metrics-server failed to resize metrics-server deployment to a size appropriate for 5k node cluster.

From pod_nanny logs (in the manual run):
Indeed, 'etcdobjectcounts' disappeared: ~ kubectl get --raw /metrics | grep etcdobjectcounts ~
It looks like etcdobjectcounts has been renamed () and this makes podnanny not able to resize metrics-server. For that reason metrics-server is not setting any resources (should be set by podnanny based on the value of the metric) which allows it to schedule on small nodes (other than the "heapster" one) and OOM continuously breaking other components running on that node (e.g. daemonset pods). The questions here: how is it possible that we discover this now as the has been merged ~20 days ago. Is it intentional that etcdobjectcounts disappears in the same version as the new metric appears? It may be hard to adjust addon-resizer if there is no grace period when both are available.
/cc Is it intentional that etcdobjectcounts disappears in 1.21 version? Shouldn't we move this to at least 1.22 to give some grace period when both metrics are available?
I can't remember, did we delete the original? We should probably retain it for a release or two prior for the reason above.
cc
Are you sure they are gone?
We are double writing at the moment.
Ah actually I think the deprecated version was incorrect, so the metric was hidden a release too soon.
/reopen Thanks for a fix! I would like to keep it open until we see at least a single green run to double check that this was the only issue in the test
Reopened this issue. <detailsIn response to : /reopen I would like to keep it open until we see at least a single green run to double check that this was the only issue in the test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
https://gke-scalability- has passed In kube-scheduler logs I see that metrics-server was correctly upsized by addon-manager (resizes happens on the very beginning of the test + hash in the name changes which means that deployment's spec has changed).
The issue seems to be resolved. I triggered another manual run in the internal project to make sure that it's not flake. We expect result in ~4h.


Question: go_test( name = "go_default_test", <del> srcs = ["yaml_test.go"], </del> <ins> srcs = [ "meta_test.go", "yaml_test.go", ], </ins> data = glob(["testdata/**"]), embed = [":go_default_library"], deps = [ <ins> "//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/yaml:go_default_library", "//vendor/sigs.k8s.io/yaml:go_default_library", ],

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add an external MetaFactory that allows for serializing multiple YAML (or JSON) files in kubeadm. The logic of marshalling YAML should be factored out from to . Why is this needed: This will allow processing component configs containing multiple documents per file. Additionally, an out-of-tree library would allow sharing this functionality with other components.
I will try work on this as carry-on from /sig cluster-lifecycle /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: register("storage-version-gc", startStorageVersionGCController) } if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.DynamicResourceAllocation) { <del> controllers["resource-claim-controller"] = startResourceClaimController </del> <ins> register("resource-claim-controller", startResourceClaimController) </ins> } return controllers

I am a new guy to kubernetes world, and today I am looking at the source code k8s, I found out a line of code can optimize in this line, `, cus the logic here should be like impl the startController logic and then we put it into the register to find out if the controller was registered twice, if so the program will panic. I think it's better that we change to the one that I provide, which can make code clean as well, but as I'm a new guy, I need some advice on it, can someone help me out? /help wanted /area controller-manager
/sig api-machinery
/sig node Would you submit a PR to fix it if you think this is a problem? /cc as the owner of DynamicResourceAllocation was as alpha level feature in v1.26.
you're say me or the pohly? this is my first time contributing on open source, might have some issues on communicate
You. Feel free to submit PR if this is an optimization. I as he is the feature owner.
thanks, I will submit PR! so the following step should be: me fork the k8s repo to my github and git pull to my local, optimize code, then git push and submit PR, right? the editing should be written on branch master
/triage accepted


Question: return []GeneratorParam{ {"name", true}, {"docker-username", true}, <del> {"docker-email", true}, </del> <ins> {"docker-email", false}, </ins> {"docker-password", true}, {"docker-server", true}, }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When I create a docker registry secret, the is also a MUST, I think we should make this parameter as optional as there is no need always put when creating a docker registry secret. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
Agree it should not be required
Did this change get reverted? I do not see as optional on stable.
I feel that this code should be changed because "--docker-email flag" is no longer required.
error: flag docker-email is required. On kubenetes 1.10.3, the above error message still exists
The field is still required. This is very frustrating as does not require an email but Kubernetes does? Is there a good reason for this?
Why is this closed? As far as I can see it is still required?


Question: // request delay without retry. // It supports wrapped errors and returns false when the error is nil. func SuggestsClientDelay(err error) (int, bool) { <del> if t := APIStatus(nil); errors.As(err, &t) && t.Status().Details != nil { </del> <ins> t, ok := err.(APIStatus) if (ok || errors.As(err, &t)) && t.Status().Details != nil { </ins> switch t.Status().Reason { // this StatusReason explicitly requests the caller to delay the action case metav1.StatusReasonServerTimeout:

I have a controller that creates objects. It uses to notice when create fails due to a deleting namespace. This check was working well until I moved it away from each call. In some cases, the error is wrapped using causing the function to always return false. I expected to work on both wrapped and unwrapped errors, like the functions do. This short program shows the issue: I am able to work around it by first calling . This type assertion should be replaced with a call to : /sig api-machinery v1.22.5, but I also see the affected code in master. <details</detailsn/a n/a k3s n/a n/a
/cc /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: // ExtraArgs are extra arguments provided to the etcd binary // when run inside a static pod. <ins> // A key in this map is the flag name as it appears on the // command line except without leading dash(es). </ins> ExtraArgs map[string]string // ServerCertSANs sets extra Subject Alternative Names for the etcd server signing cert.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading the doc about NodeRegistration, with an interest in specifying some . I noticed that the doc (e.g., ) does not tell me exactly how to identify each arg: exactly as on the command line (including leading dash(es)), as on the command line but without leading dash(es), or camelCase. I expected the doc to enable me to write some with confidence. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
FWIW the answer is flag names as on the command name but without any leading dashes. This also applies to controllerManager.extraArgs, scheduler.extraArgs, etc. which hopefully should all cover this similarly (if not now, after fixing ...)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading about how to use kubeadm and noticed a typo in the API documentation. speaks of the "node subnet" where I think "pod subnet" is intended. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/area kubeadm


Question: expectedCtx := config.CurrentContext expectedCluster := config.Contexts[expectedCtx].Cluster currentCtx := currentConfig.CurrentContext <ins> if currentConfig.Contexts[currentCtx] == nil { return errors.Errorf("failed to find CurrentContext in Contexts of the kubeconfig file %s", kubeConfigFilePath) } </ins> currentCluster := currentConfig.Contexts[currentCtx].Cluster <ins> if currentConfig.Clusters[currentCluster] == nil { return errors.Errorf("failed to find the given CurrentContext Cluster in Clusters of the kubeconfig file %s", kubeConfigFilePath) } </ins> // If the current CA cert on disk doesn't match the expected CA cert, error out because we have a file, but it's stale if !bytes.Equal(currentConfig.Clusters[currentCluster].CertificateAuthorityData, config.Clusters[expectedCluster].CertificateAuthorityData) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: [kubeconfig] Using existing kubeconfig file: "" panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x116df4b] What you expected to happen: should pass OK, or return an message describing what's wrong. How to reproduce it (as minimally and precisely as possible): run Anything else we need to know?: My cluster failed due to expired certs, I followed to recover it, but /etc/kubernetes/bootstrap- got MIA - so I copied it from another node. also got missing, so I copied that from after that I tried which panics. Environment: Kubernetes version (use ): kubectl version Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} error: You must be logged in to the server (the server has asked for the client to provide credentials) Cloud provider or hardware configuration: bare-metal OS (e.g: ): cat /etc/os-release NAME="Ubuntu" VERSION="18.04.2 LTS (Bionic Beaver)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 18.04.2 LTS" VERSIONID="18.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" PRIVACYPOLICYURL="" VERSIONCODENAME=bionic UBUNTUCODENAME=bionic Kernel (e.g.

): uname -a Linux 4.15.0-51-generic -Ubuntu SMP Wed May 15 14:27:21 UTC 2019 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Network plugin and version (if this is a network-related bug): calico Others:
It seem dereferencing clientcmdapi.Config caused the panic in validateKubeConfig
Under branch origin/release-1.4, I don't seem to find
/remove-sig node /sig cluster-lifecycle Just updating your sigs, which I think will help get the proper eyes on this the fastest :)
Relates to
i don't think it's related to: it was about a feature/documentation that is already in 1.15. ideally we should not panic in these lines: or panicing on line 242 suggests that your kubeconfig is lacking the proper information for the current cluster context. but this means that you have a corrupted kubeconfig file, that kubeadm expects to be machine generated. try adding to see which file is the problem one and share it.
/assign
/remove-kind bug /triage support
/kind feature for not panicing on machine generated kubeconfig.
Sorry - can't provoke this any longer as I had to get my cluster on it's feet. What I did was to delete the config files and let kubeadm regenerate them, which made the panic go away.


Question: // This assumes proxier.mu is locked. func (proxier *Proxier) stopProxyInternal(service proxy.ServicePortName, info *serviceInfo) error { delete(proxier.serviceMap, service) <ins> info.setAlive(false) </ins> err := info.socket.Close() port := info.socket.ListenPort() proxier.proxyPorts.Release(port)

By looking why sometime my sites magicaly slow down for a short period of time I got that : each 5 seconds dns query are jumping from < 2 ms to that . until [[ 0 -eq 1 ]]; do dig mysql-change | grep -E "time: [0-9]{3,}" && date; done ; Query time: 2187 msec Wed Jul 29 13:39:31 UTC 2015 ; Query time: 2114 msec Wed Jul 29 13:39:36 UTC 2015 ; Query time: 1890 msec Wed Jul 29 13:39:41 UTC 2015 ; Query time: 1874 msec Wed Jul 29 13:39:46 UTC 2015 ; Query time: 1930 msec Wed Jul 29 13:39:51 UTC 2015 ; Query time: 1942 msec Wed Jul 29 13:39:56 UTC 2015 ; Query time: 2350 msec Wed Jul 29 13:40:01 UTC 2015 ; Query time: 2469 msec Wed Jul 29 13:40:06 UTC 2015 ; Query time: 1898 msec Wed Jul 29 13:40:11 UTC 2015 ; Query time: 2083 msec Wed Jul 29 13:40:16 UTC 2015 ; Query time: 2110 msec Wed Jul 29 13:40:21 UTC 2015 ; Query time: 1921 msec Wed Jul 29 13:40:26 UTC 2015 ; Query time: 2023 msec Wed Jul 29 13:40:31 UTC 2015 ; Query time: 1836 msec Wed Jul 29 13:40:36 UTC 2015 ; Query time: 2182 msec Wed Jul 29 13:40:41 UTC 2015 ; Query time: 2079 msec Wed Jul 29 13:40:46 UTC 2015 ; Query time: 2066 msec Wed Jul 29 13:40:51 UTC 2015 ; Query time: 1905 msec Wed Jul 29 13:40:56 UTC 2015 ; Query time: 1999 msec Wed Jul 29 13:41:01 UTC 2015 ;

Query time: 1855 msec Wed Jul 29 13:41:06 UTC 2015 ; Query time: 2043 msec Wed Jul 29 13:41:11 UTC 2015 But it's not over, curl on ip gives me that : until [[ 0 -eq 1 ]]; do curl -I 10.0.126.115:9200 -s -w '%{time_total}' -o /dev/null | grep -E "^[1-9]" && date; done 083 Wed Jul 29 13:50:16 UTC 2015 247 Wed Jul 29 13:50:21 UTC 2015 874 Wed Jul 29 13:50:26 UTC 2015 019 Wed Jul 29 13:50:31 UTC 2015 006 Wed Jul 29 13:50:36 UTC 2015 718 Wed Jul 29 13:50:41 UTC 2015 954 Wed Jul 29 13:50:46 UTC 2015 065 Wed Jul 29 13:50:51 UTC 2015 896 Wed Jul 29 13:50:56 UTC 2015 968 Wed Jul 29 13:51:01 UTC 2015 556 Wed Jul 29 13:51:05 UTC 2015 There is something wrong in the way kube-proxy is working/updating. It's a real source of troubles on my side.
With further analysis anything run good on docker part, checking directly the virtual ip address of docker pod always return in time! checked on coreos and default kubernetes debian image.
Open question : Why using iptables to handle alone the inter-cluster communication instead of routing? each pod has it's own ip, it doesn't suffer from iptables refresh slowness, as well as missing rule. Nodes forwarding work great and on gce, it's ok to do so. Skydns might handle pod ip as RR dns A record and all that mess will run smoother. Currently I do have dbs with replication, php with some dependancies to other pods, so you can easily understand how high I'm dependant to a clean and really fast network.

I've got ~150 services wich lead to 2s with complete network stale, with 375 services it would take the whole refresh loop, and with more services what would happens? I've got the feeling that generating a complete ipfile and injecting with iprestore will do less harm. It's somehow important that kube team is looking at that. Production ready isn't really true on my case.
There is not enough information here to properly address your issue. What Kubernetes version, what OS, what cloud, how did you do setup, what network overlay if any, etc. Running on my own cluster, for example: and
you're not running enough samples, to be sure to have a least one pass on kube-proxy refresh. You'll have to run more than 5000 loops. That's why I filtered on time 1s. Good points on DNS RR. But it's the easiest way to handle multiple endpoints while real balancing is WIP. So I'm on a full 1.0.1 cluster with 1 master, 12 nodes : 11 coreos, 1 debian. The gce nodes are all on same power : n1-highmem-2. The installation is from a kube-up on release 1.0.1 package. 2 Modifications has been made on kubelet pkg to fit my need : a relocation of safeformatand_mount shell script and a safe parameter on iptables (-w switch) and that's all for coreos. The debian node is full stock, "only" there to perform debug tasks and analysis as I send here. The debian master is full stock too except the installation of ... htop . The tests provided is from the debian node, as I know it is the main OS. I've got: 104 pods (2 kubedns pods) 102 rc 101 svc 19 namespaces 17 external lb. No overlays , nothing special, I stood as close as possible to upstream, even on the coreos installation. We are not live now, so I still have time to debug anything you want and test whatever would make differences.
We know kube-proxy has a long-tail problem, but it's not usually multi-second. See the "histogram" below. Your point about doing iptables-restore in one shot is taken. root for i in $(seq 1 );

389 2 0.391 1 0.960 , bnprss wrote:
I was thinking more on iptables-restore, and I don't really see how to use it safely. There's no compare-and-swap optimistic concurrency mode and there's no way I can see to load just a single chain - only whole tables. This makes it very awkward to use for our "check if it exists and if not add it" mode, especially if anyone else is using the NAT table. Consider: kube-proxy reads state of -t nat kube-proxy iterates over state, checking for each rule it needs user modifies -t nat kube-proxy needs to make a change, replaces the -t nat section for just its own chain kube-proxy restores the state the user change is wiped out , Tim Hockin wrote:
I made some test, and I'm now sure: no traffic is handled at all while kube-proxy iptables loop, checked by logging on level 5 . I see three potential solutions : 1) keep the run whatever it cost Advantages: little changes Drawback: It will be a major problem with a lot of services, the check will take much more time than the syncloop allow. might be a huge CPU consummer 2) change the way kube-proxy check and insert rules step one : iptables-save step two : check rule existance step three : iptables inclusion of missing rule (like it does right now) Advantages: limit the rate of external program call, do the thing safely and keep custom rules by not using iptables-restore. I guess there will be some perf improvement by just parsing/checking one file. Drawback: none? 3) use external tool I've got in mind haproxy Advantages: All logic in the roadmap (and more) is allready implemented and well tested. Performance is really good. Drawback: dynamic inclustion/deletion of things must trig a reload of a map file... Network transaction shouldn't be stop anytime in the kube-proxy process.
I'll post the following PR that cover the 1st case : time to check : for i in $(seq 1 ); do curl -Ik 10.0.249.163 -s -w '%{time_total}' -o /dev/null; echo;

A better design would be store an atomic value in the ServiceInfo struct and simply check that value in each ProxyLoop(). I'm going to re-title this bug to reflect the problem , bnprss wrote:
It is really a simple ugly workaround focused on the big loop that takes ages to complete, I agree with that. And I assume it's mitigation role. My knowledge of go wont't help for your proposal, as you might have understood, it was the first time I code in that language and as a SysAdmin my skills are also limited in development tasks. So I can't handle such logic changes in an efficient way. Could you take care of it?
I'll see if I can throw something together. Do you want me to take over the lock mitigation too, or do you want to do that? It should be a relatively simple change given that you already found the place to do that.
No, it will be ok for the lock mitigation I figured out why red light cames from shippable and travis. And will comment on the PR. Thank you for taking care of the mid term solution.


Question: }, }, { <del> pod:           newResourcePod(schedulercache.Resource{MilliCPU: 1, Memory: 1, StorageOverlay: 10}), emptyDirLimit: 15, storageMedium: v1.StorageMediumMemory, </del> <ins> pod:           newResourcePod(schedulercache.Resource{}), emptyDirLimit: 25, storageMedium: v1.StorageMediumDefault, </ins> nodeInfo: schedulercache.NewNodeInfo( newResourcePod(schedulercache.Resource{MilliCPU: 2, Memory: 2, StorageOverlay: 5})), <del> fits: true, </del> <ins> fits: false, </ins> test: "storage scratchrequest exceeds allocatable", reasons: []algorithm.PredicateFailureReason{ NewInsufficientResourceError(v1.ResourceStorageScratch, 25, 5, 20), }, }, <ins> { pod:           newResourcePod(schedulercache.Resource{MilliCPU: 1, Memory: 1, StorageOverlay: 10}), emptyDirLimit: 15, storageMedium: v1.StorageMediumMemory, nodeInfo: schedulercache.NewNodeInfo( newResourcePod(schedulercache.Resource{MilliCPU: 2, Memory: 2, StorageOverlay: 5})), fits: true, test: "pod fit with memory medium", }, </ins> } for _, test := range storagePodsTests {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): local isolation Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Bug Kubernetes version (use ): kubernetes HEAD Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: For local storage isolation, overlay resource predicates for pod with only overlay or scratch storage request doesn't work. What you expected to happen: If a pod requests only overlay or scratch storage, then local storage isolation feature should limit the pods. How to reproduce it (as minimally and precisely as possible): Run local cluster from master branch, and create following pods (against a machine with less overlay fs than 1024Gi). Anything else we need to know:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig storage
CC
Fix is ready:


Question: return nil, errNotFound } <del> var matchingObjs []runtime.Object for _, obj := range objs { acc, err := meta.Accessor(obj) if err != nil { return nil, err } if acc.GetNamespace() != ns { continue } if acc.GetName() != name { continue } matchingObjs = append(matchingObjs, obj) } if len(matchingObjs) == 0 { </del> <ins> matchingObj, ok := objs[types.NamespacedName{Namespace: ns, Name: name}] if !ok { </ins> return nil, errNotFound } <del> if len(matchingObjs) > 1 { return nil, fmt.Errorf("more than one object matched gvr %s, ns: %q name: %q", gvr, ns, name) } </del> // Only one object should match in the tracker if it works // correctly, as Add/Update methods enforce kind/namespace/name // uniqueness. <del> obj := matchingObjs[0].DeepCopyObject() </del> <ins> obj := matchingObj.DeepCopyObject() </ins> if status, ok := obj.(*metav1.Status); ok { if status.Status != metav1.StatusSuccess { return nil, &errors.StatusError{ErrStatus: *status}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I used to fake a clientset to test my code. I found that it became very slow to init the clientset with a big number of objects when I was trying to simulate a large scale cluster, and the time seemed exponential increasing. For example: Initializing a clientset with: 5000 objects took 300ms objects took 2.58s objects took 11.82s objects took 54.64s. It's because the fake clientset used a slice to store each kind of objects, so it's quite slow to init the clientset with massive objects because it checked existence of an object by traversing all objects and comparing with them before adding one, which leads to O(n^2) time complexity. What you expected to happen: The fake clientset is very useful for development, it should have a reasonable time complexity to allow developers to use it for large scale scenario. How to reproduce it (as minimally and precisely as possible): Run this code: You would see the time spent on each operation, like: Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.0-1.18.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I had PR that uses map to store objects, which can reduce the init time to a reasonable value (tens of ms for objects). /assign


Question: proper values. release: v1.9 file: test/e2e/common/expansion.go <ins> - testname: VolumeSubpathEnvExpansion, subpath expansion codename: '[k8s.io] Variable Expansion should allow substituting values in a volume subpath [sig-storage] [Conformance]' description: Make sure a container's subpath can be set using an expansion of environment variables. release: v1.19 file: test/e2e/common/expansion.go - testname: VolumeSubpathEnvExpansion, subpath with absolute path codename: '[k8s.io] Variable Expansion should fail substituting values in a volume subpath with absolute path [sig-storage][Slow] [Conformance]' description: Make sure a container's subpath can not be set using an expansion of environment variables when absolute path is supplied. release: v1.19 file: test/e2e/common/expansion.go - testname: VolumeSubpathEnvExpansion, subpath with backticks codename: '[k8s.io] Variable Expansion should fail substituting values in a volume subpath with backticks [sig-storage][Slow] [Conformance]' description: Make sure a container's subpath can not be set using an expansion of environment variables when backticks are supplied. release: v1.19 file: test/e2e/common/expansion.go - testname: VolumeSubpathEnvExpansion, subpath lifecycle codename: '[k8s.io] Variable Expansion should not change the subpath mount on a container restart if the environment variable changes [sig-storage][Slow] [Conformance]' description: "Verify should not change the subpath mount on a container restart if the environment variable changes 1.tvalid subpathexpr starts a container running 2.ttest for valid subpath writes 3.tcontainer restarts 4.tdelete cleanly" release: v1.19 file: test/e2e/common/expansion.go - testname: VolumeSubpathEnvExpansion, subpath test writes codename: '[k8s.io] Variable Expansion should succeed in writing subpaths in container [sig-storage][Slow] [Conformance]' description: "Verify that a subpath expansion can be used to write files into subpaths. 1.tvalid subpathexpr starts a container running 2.ttest for valid subpath writes 3.tsuccessful expansion of the subpathexpr isn't required for volume cleanup" release: v1.19 file: test/e2e/common/expansion.go - testname: VolumeSubpathEnvExpansion, subpath ready from failed state codename: '[k8s.io] Variable Expansion should verify that a failing subpath expansion can be modified during the lifecycle of a container [sig-storage][Slow] [Conformance]' description: Verify that a failing subpath expansion can be modified during the lifecycle of a container. release: v1.19 file: test/e2e/common/expansion.go </ins> - testname: Pod events, verify event from Scheduler and Kubelet codename: '[k8s.io] [sig-node] Events should be sent by kubelets and the scheduler about pods scheduling and running  [Conformance]'

Is this a BUG REPORT or FEATURE REQUEST?: /kind api-change /sig storage /sig node /sig testing Description: Promote e2e tests for VolumeSubpathEnvExpansion to Conformance as it is GA approved. Related Issues: GA Upgrade: Feature Request: KEP:
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } for _, rs := range rss { if rsToDelete.RealServer.Equal(rs) { <del> if rs.ActiveConn != 0 { </del> <ins> // Delete RS with no connections // For UDP, ActiveConn is always 0 // For TCP, InactiveConn are connections not in ESTABLISHED state if rs.ActiveConn+rs.InactiveConn != 0 { </ins> return false, nil } klog.Infof("Deleting rs: %s", rsToDelete.String())

What happened: On pods with a lot of DNS traffic, many DNS queries to the DNS ClusterIP fail during a rolling update of the DNS pods (in our case, we use coredns). What you expected to happen: Almost all DNS queries should succeed during a rolling update. How to reproduce it (as minimally and precisely as possible): Create a coredns (this would probably be the same with kube-dns) service backed by 2 pods Access this service by creating a lot of DNS queries, for example: Wait for the IPVS connection table to contain a lot of DNS connections (a few minutes) Delete a DNS pod A significant number of queries will timeout. After 5mn (depending in the IPVS timeout configuration) everything will work ok again. Anything else we need to know?: This comes from the current logic in the graceful termination code: The current logic is to delete a RS if the number of active connections is 0. This makes sense for TCP but for UDP the number of active connections is always 0. This is an issue for DNS queries because the RS will be deleted but the IPVS connection will remain until it expires (5mn by default) and if there are a lot of DNS queries, the port will be reused and queries blackholed. In addition, will not always work here because IPVS only removes connections when a packet is processed (in the case of DNS once the client receives an answer, no more packets are sent). Environment: Kubernetes version (use ): kube-proxy OS (e.g. from /etc/os-release): Kernel (e.g. ): /area ipvs /sig network /kind bug /assign
I think there is also an issue with the way we clear the conntrack for UDP We call in : If I understand correctly this will clear conntrack entries matching and before graceful termination is done. I think we should only remove the conntrack entries when we delete the real server. In addition, we only clear the conntrack entries for <clusterIP, I think we should also clean them for Node IPs (Nodeports) and External IPs. What do you think?
I think we still need to keep this issue open even though we have merged - I assume they are different issues.

yes I think the PR is a great improvement but we probably need a few other small fixes (like the conntrack thing I mentioned)
The fix from the PR is not perfect yet for DNS traffic when qps is high (but it's definitely an improvement for UDP traffic which didn't support any kind of graceful termination). For example: Will generate 5000qps and ~500 new entries in the IPVS table per second. With a UDP timeout of 30s this leads to entries maximum. If we have 2 pods, when we delete one we have to wait for the ~7500 entries to expire, given the timeout (30s) many ports will be reused (which resets the timeout) and when the coredns pod finishes its lameduck period, we lose queries. 5000 qps against 2 coredns pods is high and unlikely but to decrease the probability of DNS queries failing, we can either: decrease the UDP timeout (in this example if set to 10s, we dont lose a single packet) increase the lameduck period (I tested with UDP timeout 30s and lameduck 180s and it worked great) (of course with more pods the impact is less important: what matters is the number of ports assigned to Terminating pods) I think being able to control the UDP timeout is very important, the default (300s) is very high (see: ) (I forked dnstrace to add the parameter: , I'll try and get it merged in the main repository)
can you please help confirm our issue? in our system, our dns query is quite often, like 1 query per 10 seconds. in this case, we found that if we restart coredns, the old IPVS rules are never deleted on some of our nodes, which caused a lot of failed dns queries. is this because the graceful termination mechanism introduced here? In order to delete the RS and remove the ipvs rules, we have to wait the DNS query expire, but before the old dns query expires, new dns query came in, so grace termination will never see and old ipvs rules are never cleaned up, causing dns query issue.

if this is the case, I would suggest to introduce a force termination mechanism
: it depends how you query DNS: if you have a persistent resolution socket, the IPVS backend may never expire but in that case your resolver should timeout and reconnect which will use an active backend. If you use a standard resolver (glibc for instance) each query will get a new connection so you will reach an active coredns unless you query frequently enough so you end up reusing a same source port associated with a removed backend (but below 100 queries/second this is very unlikely). I have ideas to address both use cases: configure more aggressive timeouts (in particular for UDP) and make them configurable (see issue: ) forcefully remove real servers (coredns in your case) that are still tracked by IPVS after X minutes (today we only delete them when they have 0 active and 0 inactive connections which may never happen depending on your workload) may have ideas on this topic too
DNS resolution through kube-proxy is tricky for several reasons regardless of the proxy mode (for iptables for instance, many people have conntrack issues when doing a lot of queries). In our case, we are experimenting with node local coredns caches (you can also have a look at which is an implementation of this idea)
isn't 100 queries/second quite common for a site? QPS can be thousands nowadays. we did not do special configuration for our dns so it should be using glibc. I think what you mentioned about reusing the same source port should be our issue here. by checking the output of ipvsadmin, it seems the source port is reused for different udp queries <img width="517" alt="" src="https://user-"do you have any other suggestions on this issue?
btw, forcefully remove real servers is what I want here.
I meant 100 qps for a single client (which can definitely happen though) Yes it is very likely that you have the issue I mentioned Until we implement the solution I mentioned natively, you can try changing the ipvs udp timeout on your instances. The default are: You can change udp timeout to 30s (or even lower depending on your connection rate): (0 here means do not touch tcp/tcpfin)


Question: }, err:  "spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0]: Invalid value:", test: "empty NodeSelectorTerm (selects nothing) cannot become populated (selects something)", <ins> }, { old: core.Pod{ Spec: core.PodSpec{ Affinity:        nil, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{{ MatchExpressions: []core.NodeSelectorRequirement{{ Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }}, }}, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, test: "nil affinity can be mutated for gated pods", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity:        nil, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{{ MatchExpressions: []core.NodeSelectorRequirement{{ Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }}, }}, }, }, PodAffinity: &core.PodAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: []core.PodAffinityTerm{ { TopologyKey: "foo", LabelSelector: &metav1.LabelSelector{ MatchLabels: map[string]string{"foo": "bar"}, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "pod updates may not change fields other than", test: "the podAffinity cannot be updated on gated pods", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity:        nil, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{{ MatchExpressions: []core.NodeSelectorRequirement{{ Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }}, }}, }, }, PodAntiAffinity: &core.PodAntiAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: []core.PodAffinityTerm{ { TopologyKey: "foo", LabelSelector: &metav1.LabelSelector{ MatchLabels: map[string]string{"foo": "bar"}, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "pod updates may not change fields other than", test: "the podAntiAffinity cannot be updated on gated pods", </ins> }, } for _, test := range tests {

When a gated pod is created with nil affinity and a user/controller tries to update its nodeAffinity, the validation fails. The pod gets updated with the new affinity as per rules in a pod as in the following using : to update the pod as with: No response <details</details<detailsTested on kind and OpenShift </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling


Question: ContainersToKill:         getKillMapWithInitContainers(basePod, baseStatus, []int{}), }, }, <ins> "Pod sandbox not ready, init container failed, and RestartPolicy == OnFailure; create a new pod sandbox": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyOnFailure }, mutateStatusFn: func(status *kubecontainer.PodStatus) { status.SandboxStatuses[0].State = runtimeapi.PodSandboxState_SANDBOX_NOTREADY status.ContainerStatuses[2].ExitCode = 137 }, actions: podActions{ KillPod:                  true, CreateSandbox:            true, SandboxID:                baseStatus.SandboxStatuses[0].Id, Attempt:                  uint32(1), NextInitContainerToStart: &basePod.Spec.InitContainers[0], ContainersToStart:        []int{}, ContainersToKill:         getKillMapWithInitContainers(basePod, baseStatus, []int{}), }, }, </ins> } { pod, status := makeBasePodAndStatusWithInitContainers() if test.mutatePodFn != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For a Pod with , kubelet would attempt to create a new sandbox for it even after all container succeeded. Logs as below: This issue is similar to which fixed the case of pods whose RestartPolicy are Never. It caused unnecessary CRI and CNI calls, confusing logs and conflicts between the routine that creates the new sandbox and the routine that kills the Pod. It's likely also the root cause of because sometimes the routine that creates the second sandbox and the routine that kills unwanted pod run concurrently, then the netns would be gone when calling CNI. What you expected to happen: kubelet shouldn't create a new sandbox for pod with RestartPolicyOnFailure if all containers succeeded How to reproduce it (as minimally and precisely as possible): Create a pod like this: Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Created which may fix this.
/assign


Question: name: "Change image for multiple containers with set-image", opts: &DebugOptions{ CopyTo:    "myapp-copy", <del> Container: "app", </del> SetImages: map[string]string{"*": "busybox", "app": "app-debugger"}, }, havePod: &corev1.Pod{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using without specifying should modify existing container images without trying to create a new one. Instead it tries to create a new container and returns an error that isn't set: What you expected to happen: This command should change all container images to instead of returning an error. /sig cli /kind bug /priority important-soon
/assign /triage accepted
My kubernetes cluster version is 1.19.0,and i use kubectl-debug version 0.1.1 try to debug my pod,bug i got error: <img width="1470" alt="image" src="https://user-"Who one know about that


Question: t.Logf("Created service: %s", svcNodePort.Name) } <ins> func TestServicesFinalizersPatchStatus(t *testing.T) { serviceCIDR := "10.0.0.0/16" clusterIP := "10.0.0.21" nodePort := 30443 _, ctx := ktesting.NewTestContext(t) ctx, cancel := context.WithCancel(ctx) defer cancel() client, _, tearDownFn := framework.StartTestServer(ctx, t, framework.TestServerSetup{ ModifyServerRunOptions: func(opts *options.ServerRunOptions) { opts.ServiceClusterIPRanges = serviceCIDR }, }) defer tearDownFn() for _, testcase := range []string{"spec", "status"} { t.Run(testcase, func(t *testing.T) { // Create a NodePort service with one finalizer svcNodePort := v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:       "svc" + testcase, Finalizers: []string{"foo.bar/some-finalizer"}, }, Spec: v1.ServiceSpec{ ClusterIP: clusterIP, Ports: []v1.ServicePort{{ Port:       8443, NodePort:   int32(nodePort), TargetPort: intstr.FromInt32(8443), Protocol:   v1.ProtocolTCP, }}, Type: v1.ServiceTypeNodePort, }, } ns := framework.CreateNamespaceOrDie(client, "test-service-finalizers-"+testcase, t) defer framework.DeleteNamespaceOrDie(client, ns, t) // Create service if _, err := client.CoreV1().Services(ns.Name).Create(ctx, &svcNodePort, metav1.CreateOptions{}); err != nil { t.Fatalf("unexpected error creating service: %v", err) } t.Logf("Created service: %s", svcNodePort.Name) // Check the service has been created correctly svc, err := client.CoreV1().Services(ns.Name).Get(ctx, svcNodePort.Name, metav1.GetOptions{}) if err != nil || svc.Spec.ClusterIP != clusterIP { t.Fatalf("created service is not correct: %v", err) } t.Logf("Service created successfully: %+v", svc) // Delete service if err := client.CoreV1().Services(ns.Name).Delete(ctx, svcNodePort.Name, metav1.DeleteOptions{}); err != nil { t.Fatalf("unexpected error deleting service: %v", err) } t.Logf("Deleted service: %s", svcNodePort.Name) // Check that the service was not deleted and the IP is already allocated svc, err = client.CoreV1().Services(ns.Name).Get(ctx, svcNodePort.Name, metav1.GetOptions{}) if err != nil || svc.Spec.ClusterIP != clusterIP || int(svc.Spec.Ports[0].NodePort) != nodePort || svc.DeletionTimestamp == nil || len(svc.ObjectMeta.Finalizers) != 1 { t.Fatalf("Service expected to be deleting and with the same values: %v", err) } t.Logf("Service after Delete: %+v", svc) // Remove the finalizer updated := svc.DeepCopy() updated.ObjectMeta.Finalizers = []string{} patchBytes, err := getPatchBytes(svc, updated) if err != nil { t.Fatalf("unexpected error getting patch bytes: %v", err) } if testcase == "spec" { if _, err = client.CoreV1().Services(ns.Name).Patch(ctx, svcNodePort.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}); err != nil { t.Fatalf("unexpected error removing finalizer: %v", err) } } else { if _, err = client.CoreV1().Services(ns.Name).Patch(ctx, svcNodePort.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status"); err != nil { t.Fatalf("unexpected error removing finalizer: %v", err) } } t.Logf("Removed service finalizer: %s", svcNodePort.Name) // Check that the service was deleted _, err = client.CoreV1().Services(ns.Name).Get(ctx, svcNodePort.Name, metav1.GetOptions{}) if err == nil { t.Fatalf("service was not delete: %v", err) } // Try to create service again without the finalizer to check the ClusterIP and NodePort are deallocated svc = svcNodePort.DeepCopy() svc.Finalizers = []string{} if _, err := client.CoreV1().Services(ns.Name).Create(ctx, svc, metav1.CreateOptions{}); err != nil { t.Fatalf("unexpected error creating service: %v", err) } // Delete service if err := client.CoreV1().Services(ns.Name).Delete(ctx, svc.Name, metav1.DeleteOptions{}); err != nil { t.Fatalf("unexpected error deleting service: %v", err) } }) } } </ins> // Regresion test for https://issues.k8s.io/115316 func TestServiceCIDR28bits(t *testing.T) { serviceCIDR := "10.0.0.0/28"

Why PatchService method is used for deleting finalizers in cloud-controller-manager, This method updates with service/status, It seems to go unnoticed by the API server, resulting in the API server being unaware when the finalizer field on the service is deleted, leading to a delay in the release of service-related NodePorts. service nodeport be released promptly when service is deleted. create a loadblancer service with nodeport seted,then delete it and recreate. cloud-controller-manager add a finalizer on service. delete this service and then cloud-controller remove finalizer. but nodeport is not released when service deleted. ref i change like this, it seems like releasing nodeport when service is deleted : No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
Are you sure? that method updates the status and remove the finalizer ... otherwise , who removes the finalizer?
I apologize for any confusion. What I meant was that the update of the status and the removal of the finalizer are performed through updating service/status. Therefore, removing the finalizer does not release resources.It seems like removal of the finalizer by service/status, store.AfterDelete is nil. i think remove the finalizer should by patch servcie and updates the status should by patch service/status.Now we patch both by servcie/status.
no need to apologize, let me see if I can reproduce this locally first, Services api allocation is complex , it overrides the standard AfterDelete and release all allocated resources there /assign
/sig network ack the problem
will look for a possibly-related past cloud provider issue.
this will be fixed by , it is a problem in the apiserver logic
Cool! I forgot that you can patch finalizers thru /status. I repro'ed it on both ClusterIP and NodePort.

Saw this in the logs: So the repair loop, yet again, saves our butt but also hides a bug.
we need metrics here about to have visibility of the repairs
I also found this problem in versions 1.23 and 1.27
yep, I started the backports , but we can only backport to the supported versions, 1.23 and 1.24 are EOL


Question: stats, err := provider.ListPodCPUAndMemoryStats() assert := assert.New(t) assert.NoError(err) <del> assert.Equal(4, len(stats)) </del> <ins> assert.Equal(5, len(stats)) </ins> podStatsMap := make(map[statsapi.PodReference]statsapi.PodStats) for _, s := range stats {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: CPU.Time is set to cs.StartTime but ps.Memory.Time is set to timestamp when collecting the memoryusage. What you expected to happen: CPU.Time is set to cs.CPU.Time How to reproduce it (as minimally and precisely as possible): Check the code Anything else we need to know?: Environment: Kubernetes version (use ): master branch when creating this issue. Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig kubelet
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig kubelet Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I am happy to take over this. /assign


Question: }) }) <ins> framework.KubeDescribe("Kubectl get componentstatuses", func() { It("should get componentstatuses", func() { By("getting list of componentstatuses") output := framework.RunKubectlOrDie("get", "componentstatuses", "-o", "jsonpath={.items[*].metadata.name}") components := strings.Split(output, " ") By("getting details of componentstatuses") for _, component := range components { By("getting status of " + component) framework.RunKubectlOrDie("get", "componentstatuses", component) } }) }) </ins> framework.KubeDescribe("Kubectl apply", func() { It("should apply a new configuration to an existing RC", func() { controllerJson := commonutils.SubstituteImageName(string(readTestFileOrDie(redisControllerFilename)))

<!-- Please only use this template for submitting enhancement requests --What would you like to be : e2e test suites for: GET /api/v1/componentstatuses/{name} Why is this needed: The above API is not performed in current e2e test. By adding test for it, e2e test coverage is improved. /sig testing


Question: } defer os.RemoveAll(basePath) kb := &Kubelet{ <del> rootDirectory:    basePath, </del> <ins> rootDirectory:    filepath.Clean(basePath), </ins> recorder:         &record.FakeRecorder{}, cadvisor:         cadvisor, nodeLister:       testNodeLister{},

pull-kubernetes-unit, see example failure in It appears to be a new test in No response No response No response /sig
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Repro's locally:
/cc can you PTAL? It seems related to new test in
flaking regularly -
actually, I'm not sure this is a test-only race... it looks like self mutates , and that is called in a goroutine in parallel with other methods that access rootDirectory: I suspect this has always been a race, and the unit test is exercising / catching it more
moving the filepath.Clean to init time: and changing the check in setupDataDirs to be a validation rather than a mutation: might resolve the immediate race
/assign
is open to fixing it. The data race we find here should be fixed. However, is another one that we need to fix. I add a new commit to fix that as well.
did you see that race as well or did you just see the issue opened? that was in a code path that should be feature gated so I wouldn't have expected it to trigger in the kubemark unit tests
I may mixed problems together. Yes. will not trigger in the kubemark unit tests. Using stressing, I find the data race cannot fix the problem completely. Then I saw and cherry-picked the fix to my PR. But the hollowNode test still flakes and I looked into it and find it is the volumeManager problem. is just another potential data race in .

the rootDirectory and volumeManagerSyncTime race fixes are easy to reason about and fix code that looks like it could really race in currently enabled-by-default code looks like a real race in disabled-by-default code. I'll defer to kubelet folks and whether to mix that fix into
I think I should drop that fix. This will make things more clear.


Question: {Group: "storage.k8s.io", Version: "v1beta1"}:                {group: 16800, version: 9}, {Group: "apiextensions.k8s.io", Version: "v1beta1"}:          {group: 16700, version: 9}, {Group: "admissionregistration.k8s.io", Version: "v1alpha1"}: {group: 16700, version: 9}, <ins> {Group: "scheduling.k8s.io", Version: "v1alpha1"}:            {group: 16600, version: 9}, </ins> } func apiServicesToRegister(delegateAPIServer genericapiserver.DelegationTarget, registration autoregister.AutoAPIServiceRegistration) []*apiregistration.APIService {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: According to the two changes are required to enable it: Add to apiserver and scheduler. Add to apiserver. So I modified the apiserver manifest: And the scheduler manifest: confirms that both are running with the required changes. However, I cannot create PodPriorities by following the example in the documentation: also doesn't think the API is enabled: The only relevant message in the apiserver log is: What you expected to happen: I expected to be able to create podpriority resources by following the example in the documentation. How to reproduce it (as minimally and precisely as possible): See description. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): CentOS Linux 7.4 (1708) Kernel (e.g. ): Linux master-head-1 4.13.4- SMP Wed Sep 27 13:32:23 EDT 2017 x8664 x8664 x8664 GNU/Linux Install tools: Custom scripts Others:
/sig scheduling
The example yaml of priorityclass is wrong, the right version is
please try the recommendation from and close this issue?
unfortunately that version change to the YAML does not fix the problem for me either: To me it looks like the error message points me to where I don't see defined?
Yes, you are right, I have commit a pr to fix this.
Not sure but after using the updated config I am able to create Priority Class successfully but unable to use the feature while creating POD. Here are the errors observed Getting following error while creating pod: The PodPriority is enabled in both API server and Scheduler
I'm able to create PriorityClass and schedule pod with specified PriorityClass on Minikube 0.26.0. ` !
This issue was sorted - on kubeadm production setup, this flag needs to be enabled for kubelet as well - updated doc
I've created the PriorityClass and deployed the pod with the specified PriorityClass. The pod is getting created. I'm wondering is there any way to check the PriorityClass of the pod either in UI or CLI? The steps you've shown is giving me the same result. Can you please help me ? Thanks in advance.


Question: <ins> //go:build windows // +build windows /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package logs import ( "os" "syscall" ) // Based on Windows implementation of Windows' syscall.Open // https://cs.opensource.google/go/go/+/refs/tags/go1.22.2:src/syscall/syscall_windows.go;l=342 // In addition to syscall.Open, this function also adds the syscall.FILE_SHARE_DELETE flag to sharemode, // which will allow us to read from the file without blocking the file from being deleted or renamed. // This is essential for Log Rotation which is done by renaming the open file. Without this, the file rename would fail. func openFileShareDelete(path string) (*os.File, error) { pathp, err := syscall.UTF16PtrFromString(path) if err != nil { return nil, err } var access uint32 = syscall.GENERIC_READ var sharemode uint32 = syscall.FILE_SHARE_READ | syscall.FILE_SHARE_WRITE | syscall.FILE_SHARE_DELETE var createmode uint32 = syscall.OPEN_EXISTING var attrs uint32 = syscall.FILE_ATTRIBUTE_NORMAL handle, err := syscall.CreateFile(pathp, access, sharemode, nil, createmode, attrs, 0) if err != nil { return nil, err } return os.NewFile(uintptr(handle), path), nil } </ins>

The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows