Question: } } <del> func expectStop(t *testing.T, w *worker, c bool, msg string) { if c { t.Errorf("[%s - %s] Expected to stop, but did not", w.probeType, msg) } } </del> func resultsManager(m *manager, probeType probeType) results.Manager { switch probeType { case readiness:

We detected behavior change in v1.21.0 Example: POD has and for some containers Behavior in v1.20.5: starts applied startupProbe success, then start checks for livenessProbe working failed restarted startupProbe applied startupProbe success, then start checks for livenessProbe working Behavior in v1.21.0: starts applied startupProbe success, then start checks for livenessProbe working failed restarted applied, startupProbe skipped not ready yet, killed by liveness check in restart loop So, in v1.21 is applied to "fresh" starts only and not used when container restarted. There is not mentions in changelog that behavior of the is changed. should run on container restarts also. POD with and fail that doesn't apply after container restart Kubernetes version (use ):Cloud provider or hardware configuration: baremetal
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign
Thanks for the quick fix
it was fixed in master (1.22) but not for 1.21, does it need a backport? /reopen
Reopened this issue. <detailsIn response to : it was fixed in master (1.22) but not for 1.21, does it need a backport? /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close never mind i just saw the backports.
Closing this issue. <detailsIn response to : /close </details
can you summarize the impacted patch releases (looks like the regression was backported to 1.18-1.20 as well)
These 4 patch releases are impacted, 1.18.18, 1.19.10, 1.20.6, 1.21.0


Question: } e.mutex.Lock() <del> e.devices = devices </del> <ins> // NOTE: Return a copy of 'devices' instead of returning a direct reference to local 'devices' e.devices = make(map[string]pluginapi.Device) for _, d := range devices { e.devices[d.ID] = d } </ins> e.mutex.Unlock() e.callback(e.resourceName, added, updated, deleted)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area hw-accelerators /sig node What happened: When re-writing the endpoint code I discovered that the is wrong and can lead to a data race while calling the function. The current logic is a bit convoluted and would make more sense if the map was created in the for loop. However it seems like once the map is assigned to the structure, a device update while calling the function will lead to a data race, Anything else we need to know?: /assign
Not quite sure about the race scenario you described, which func do you mean by function ? The ?
Did you mean by ?  map is being assigned to structure with locking protection only.  Did you want to say that a race will happen if is being updated and at the same time gets invoked? If yes, i dont think so. At both the places, first lock is taken. So only one can run at a time and other will wait on mutex. Please let me know if i mistaken something.
Sorry I meant  though essentially the manager.Devices function calls the endpoint.getDevices(). Yes thanks for clarifying this.
I believe we all agree that there is no race in the scenario described in the description of this issue. Shall we close this issue?
there is a race, I think I didn't fully read your answer at the time sorry. Because the way maps works when the map in the structure, It is then making the Endpoint's map point to the local copy of . Which means that every time we lock in the that and thus the endpoint's map .
thanks! I got confused b/w call-by-value and call-by-reference and forgot that in golang maps are call by reference.
this possibility still remains that some devices may disappear from the endpoint, while manager knows about those.
Yes. That is the reason why in the latest commit in the manager is taking a snapshot of the devices known and based the device allocation on that. There are possibilities where devices come and go between the two points in time: See above diagram. At time the manager starts processing the device allocation requests for a Pod. It takes a snapshot of the all devices it knows. It doesn't care how many great devices appeared in history or how many devices may go dead in seconds.

The manager cannot predict that, so "let it be" is its attitude towards device changes. At time , it tries allocate device ID . So long is good at the moment, it proceeds to allocate and . If and only when all requested devices can be allocated, the Pod can be successfully created. It could be the case that device A3 has gone offline after and then come back to life soon before . That is acceptable. As a transaction, the Pod device allocation is okay. We cannot pretend that we can get well prepared for all device failures. For example, after the manager has confirmed the device allocation, may fail again. The Pod still cannot use it. But there is nothing we can do (it is life...).
The race this issue is discussing is about a "real" race where mutex locks are not handled correctly. The issue you are mentioning is a different one that explains very well in the above comment. It's a complex interaction but is currently correctly handled by the allocation and pod rejection process.


Question: metadataEmail            = metadataUrl + "instance/service-accounts/default/email" storageScopePrefix       = "https://www.googleapis.com/auth/devstorage" cloudPlatformScopePrefix = "https://www.googleapis.com/auth/cloud-platform" <del> googleProductName        = "Google" </del> defaultServiceAccount    = "default/" )

/kind bug What happened: Attempting to start kubelet on CoreOS machine with latest stable (). I have many other blades which are PXE booting where the commands work. For some reason on this box where CoreOS is installed to disk, it is not working. I have checked every possible setting and file and they are all identical ( permissions and values ). What you expected to happen: Kubelet to start and start up manifests ( api, proxy, controller ). How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Kubelet never actually starts a docker container. Doing doesn't show anything exited. I can docker pull the images from google. I've tried almost every setting I can change over the past 2 days. Environment: Kubernetes version (use ): I've tried both: v1.8.5 and v1.9.0 Cloud provider or hardware configuration: Dell R720XD. 24core 64gb ram, 22 SSDs. Most are ceph OSDs, sda is raid 1 SSDs for the OS and ideally kubernetes master. Bonded 10G cards for private network ( flannel ) and bonded 1g cards for public network. OS (e.g. from /etc/os-release): Kernel (e.g. ):Install tools: Baremetal plain install. No tools used. Others: Downloaded the bins into /opt/bin/ as described in kubernetes docs. The node is a member of etcd cluster. Manifests files are the usual, but kubelet doesn't get that far so I dont want to clutter and post configs. Systemd Kubelet Config: master- Logs from kubelet: After days of debugging, I believe it has to do with these lines in the logs: When it tries to connect to the docker-shim which it creates, on the other machines it looks like this: The success message never gets printed on the failing node. Both are identical. Flannel is up and running and can ping other flannel IPs on the network. Both docker infos are: The running containers are the OSDs.
/sig node /sig cluster-lifecycle
what are the contents of your file? looks like at least a few logs think that you are running on GCE example the log from here -
VERY interesting contents actually.

The actual device is a second hand Dell R720XD, GSA ( Google Search Appliance ) ! The contents of the file are: Yes, and even when I put it still looked for it. I spent 3 days on the issue but eventually gave up - which is rare. I tried running the kubelet in rkt, docker and just the bin on coreOS itself. No matter what it could never actually start a manifest container. It never actually reached the "kubelet started". The http port never responded either. I'd be happy to attempt it again if you think this is the cause.
wow!!! yes, if you can fix the product_name somehow, i'll help you move along :) cross my fingers
Good sir, Go0o0o0o0oood sir - after literally almost 72hrs of little to no sleep, trying every possible thing I could imagine, your simple suggestion has fixed it... You have no idea how grateful I am! lol Since rebranding the BIOS back to dell from google is a pain ( ) I am running the kubelet in rkt, and just mounting a file to that path. The just contains . Everything is up and running as expected when I started this journey lol. THANK YOU for taking the time to help.


Question: } } } <ins> // TestFixDeprecatedTaintKey verifies we have backwards compatibility after upgraded alpha taint key to GA taint key. // TODO(resouer): this is introduced in 1.9 and should be removed in the future. func TestFixDeprecatedTaintKey(t *testing.T) { fakeNow := metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC) evictionTimeout := 10 * time.Minute fakeNodeHandler := &testutil.FakeNodeHandler{ Existing: []*v1.Node{ { ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ kubeletapis.LabelZoneRegion:        "region1", kubeletapis.LabelZoneFailureDomain: "zone1", }, }, }, }, Clientset: fake.NewSimpleClientset(&v1.PodList{Items: []v1.Pod{*testutil.NewPod("pod0", "node0")}}), } nodeController, _ := newNodeControllerFromClient(nil, fakeNodeHandler, evictionTimeout, testRateLimiterQPS, testRateLimiterQPS, testLargeClusterThreshold, testUnhealthyThreshold, testNodeMonitorGracePeriod, testNodeStartupGracePeriod, testNodeMonitorPeriod, nil, nil, 0, false, true) nodeController.now = func() metav1.Time { return fakeNow } nodeController.recorder = testutil.NewFakeRecorder() deprecatedNotReadyTaint := &v1.Taint{ Key:    algorithm.DeprecatedTaintNodeNotReady, Effect: v1.TaintEffectNoExecute, } nodeNotReadyTaint := &v1.Taint{ Key:    algorithm.TaintNodeNotReady, Effect: v1.TaintEffectNoExecute, } deprecatedUnreachableTaint := &v1.Taint{ Key:    algorithm.DeprecatedTaintNodeUnreachable, Effect: v1.TaintEffectNoExecute, } nodeUnreachableTaint := &v1.Taint{ Key:    algorithm.TaintNodeUnreachable, Effect: v1.TaintEffectNoExecute, } tests := []struct { Name           string Node           *v1.Node ExpectedTaints []*v1.Taint }{ { Name: "Node with deprecated not-ready taint key", Node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ kubeletapis.LabelZoneRegion:        "region1", kubeletapis.LabelZoneFailureDomain: "zone1", }, }, Spec: v1.NodeSpec{ Taints: []v1.Taint{ *deprecatedNotReadyTaint, }, }, }, ExpectedTaints: []*v1.Taint{nodeNotReadyTaint}, }, { Name: "Node with deprecated unreachable taint key", Node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ kubeletapis.LabelZoneRegion:        "region1", kubeletapis.LabelZoneFailureDomain: "zone1", }, }, Spec: v1.NodeSpec{ Taints: []v1.Taint{ *deprecatedUnreachableTaint, }, }, }, ExpectedTaints: []*v1.Taint{nodeUnreachableTaint}, }, { Name: "Node with not-ready taint key", Node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ kubeletapis.LabelZoneRegion:        "region1", kubeletapis.LabelZoneFailureDomain: "zone1", }, }, Spec: v1.NodeSpec{ Taints: []v1.Taint{ *nodeNotReadyTaint, }, }, }, ExpectedTaints: []*v1.Taint{nodeNotReadyTaint}, }, { Name: "Node with unreachable taint key", Node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ kubeletapis.LabelZoneRegion:        "region1", kubeletapis.LabelZoneFailureDomain: "zone1", }, }, Spec: v1.NodeSpec{ Taints: []v1.Taint{ *nodeUnreachableTaint, }, }, }, ExpectedTaints: []*v1.Taint{nodeUnreachableTaint}, }, } for _, test := range tests { fakeNodeHandler.Update(test.Node) if err := syncNodeStore(nodeController, fakeNodeHandler); err != nil { t.Errorf("unexpected error: %v", err) } nodeController.doFixDeprecatedTaintKeyPass(test.Node) if err := syncNodeStore(nodeController, fakeNodeHandler); err != nil { t.Errorf("unexpected error: %v", err) } node, err := nodeController.nodeLister.Get(test.Node.GetName()) if err != nil { t.Errorf("Can't get current node...") return } if len(node.Spec.Taints) != len(test.ExpectedTaints) { t.Errorf("%s: Unexpected number of taints: expected %d, got %d", test.Name, len(test.ExpectedTaints), len(node.Spec.Taints)) } for _, taint := range test.ExpectedTaints { if !taintutils.TaintExists(node.Spec.Taints, taint) { t.Errorf("%s: Can't find taint %v in %v", test.Name, taint, node.Spec.Taints) } } } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When discussing issue in , I noticed: although is using key , it has actually been changed to in for quite a while. I am thinking we may need to fix this inconsistency in 1.9, by moving out of alpha. Anything else we need to know?: Task tracker: [x] update [x] update kubernetes doc
/sig scheduling


Question: if err != nil { t.Errorf("Unexpected error when calling GetMetrics %v", err) } <del> if e, a := (expectedEmptyDirUsage.Value() + expectedBlockSize), actual.Used.Value(); e != a { </del> <ins> if e, a := (expectedEmptyDirUsage.Value() + getExpectedBlockSize(filepath.Join(tmpDir, "f1"))), actual.Used.Value(); e != a { </ins> t.Errorf("Unexpected Used for directory with file.  Expected %v, got %d.", e, a) } }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): TestMetricsDuGetCapacity Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master branch (tip) Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): SLES 12 Kernel (e.g. ): Linux pts00449-vm23 3.12.28-4-default SMP Thu Sep 25 17:02:34 UTC 2014 () ppc64le ppc64le ppc64le GNU/Linux Install tools: Others: What happened: make test WHAT="./pkg/volume" Running tests for APIVersion: v1,apps/v1beta1,,,,,autoscaling/v1,autoscaling/v2alpha1,batch/v1,batch/v2alpha1,,extensions/v1beta1,,policy/v1beta1,,,,,,federation/v1beta1 +++ [0404 11:22:46] Running tests without code coverage --- FAIL: TestMetricsDuGetCapacity (0.01s) Unexpected Used for directory with file. Expected 4096, got .

FAIL FAIL 0.137s Makefile:164: recipe for target 'test' failed make: [test] Error 1 Test case failed What you expected to happen: Test case should pass How to reproduce it (as minimally and precisely as possible): make test WHAT="./pkg/volume" Anything else we need to know*: Found this similar issue: However the values returned are different, see this string "Expected 4096, got "
Hi, I am able to run that tests: I am running RHEL ppc64le, latest k8s with go1.8, probably something specific to do with SLES?
I still get the error on SLES, updated the code to latest and using go1.8, maybe it is SLES (or my machine?) specific talks about /tmp space, not sure if its related..
Hi , Some investigation into this i found that the issue is due to the hardcoding on line 32 in (similar from comments here: ) I changed it to remove the hardcoding (patch attached) and the test case passes on SLES as well, i am not sure if the patch is appropriate so requesting you to review it as appropriate
Hi Vaibhav, Have you raised any PR on it? If raised, reviewing would be easier. There is one thing I noticed: In the above code, you are comparing actual used value with total of empty directory usage and expected block size. When you create a new file like "f1 created here with only "Hello World", it consumes 4K in Ubuntu/RHEL. This size is same as for an empty directory. What is this size in SLES? So, either you hardcode this value(conditional for SLES) or call getExpectedBlockSize() on f1 file path instead of tmpDir.
Hi I have created the PR for the patch as suggested.. I have also changed the original patch slightly to use getExpectedBlockSize() on f1 file path instead of tmpDir as you suggested as that I think would be the right way to do it.. Thanks! Vaibhav
/sig storage


Question: ExpectedContent: expectedContent, }, } <del> volume.TestVolumeClient(f, testConfig, nil, "", tests) </del> <ins> volume.TestVolumeClientSlow(f, testConfig, nil, "", tests) </ins> } l.testCase.TestDynamicProvisioning() })

Which jobs are flaking: gce-cos-master-alpha-features Which test(s) are flaking: [sig-storage] CSI Volumes [Driver: ][Serial] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] Testgrid link: Reason for failure: It takes 6 minutes to start a pod with PVC that has been restored from a snapshot, while the test waits only 5 minutes: csi-attacher sidecar says: Perhaps the provisioner should wait until the volume is ready to be used? That may not help with this test timeout, as PVC is provisioned after the Pod is scheduled and the total time could be the same. All this is caused by my PR to test block snapshots, It reworked the tests from using . If snapshots are slow, should all persistent storage tests use / ?
/sig storage
Generally, cutting a snapshot is quick, but uploading can take longer. Does provisioner wait until the snapshot is "ready" before returning success?
cc
(But yes, we should increase the timeout because overall time will still be longer no matter which order we take)
The provisioner does not wait. If snapshot is not readyToUse, creating a volume from it will fail. I'll have to check to see if there's logic to wait in the e2e test. Regarding GCE PD, at restore time, do we need to download the snapshot from object store? If that's the case, it could take a while to download as well.
volume controller should keep trying until snapshot is ready. Looks like attacher tried to attach while disk is not ready. It is strange because I think attach only happen after volume is successfully provisioned.
AFAIK, most cloud providers' snapshot ATM will be uploaded to some remote when taking, and will be downloaded when restoring. GCE PD is one of them.
It's not taking snapshots what slows the test, it's restoring it - returns quickly, but the returned volume is not really usable and attach fails for ~4 minutes until the snapshot is fully restored somewhere in GCE. Now that I am thinking about it, maybe it's better if the driver returns the volume early, so it does not end up orphaned when user deletes corresponding PVC and the provisioner is restarted at the same time.

Anyway, I'll extend the timeout in volume checks. /assign
Which brings me to idea: what about other storage backends? Some of them may take even longer that 10 minutes to restore a snapshot. Or more than 5 minutes to provision an empty volume and run a pod. There should be some configuration of timeouts (timeout multiplier?) for
I opened to investigate from GCE side if this behavior is expected.


Question: delete(cache.assumedPods, key) delete(cache.podStates, key) default: <del> return fmt.Errorf("pod %v state wasn't assumed but get forgotten", key) </del> <ins> return fmt.Errorf("pod %v wasn't assumed so cannot be forgotten", key) </ins> } return nil }

/kind bug There are several error messages in the scheduler cache that are confusing to read, at least one of these surfaced up to end users via Events. I noticed these two in particular but there may be others: Whether or not these should actually be surfacing as events, the messages should be cleaned up.
cc
Seems that these are internal or implement details in scheduler, do our end users can fully understand what the events really mean? I think it is depend on our own implement. idea?
I believe, the problem is with in the message but I am not able to reproduce it, can you please share steps to reproduce this.
No unfortunately, one of our users reported seeing it when there system was in a bad state, not in a reproducible way. The point of the issue is just that the grammar in those messages is bad, which should be fixable even if you can't get them to fire.


Question: updated.Status.LoadBalancer = *newStatus klog.V(2).Infof("Patching status for service %s/%s", updated.Namespace, updated.Name) <del> _, err := patch(s.kubeClient.CoreV1(), service, updated) </del> <ins> _, err := servicehelper.PatchService(s.kubeClient.CoreV1(), service, updated) </ins> return err }

PR service patch methods to Once there is a release with that PR included, the code in service-controller and legacy-cloud-providers/gce can be updated to use those. /area platform/gce /assign
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Created a PR to use these new functions in Service-controllers.


Question: volumeName := "aws://" + az + "/" + awsID // apply tags <del> if volumeOptions.Tags != nil { if err := s.createTags(awsID, *volumeOptions.Tags); err != nil { </del> <ins> tags := make(map[string]string) for k, v := range volumeOptions.Tags { tags[k] = v } if s.getClusterName() != "" { tags[TagNameKubernetesCluster] = s.getClusterName() } if len(tags) != 0 { if err := s.createTags(awsID, tags); err != nil { </ins> // delete the volume and hope it succeeds _, delerr := s.DeleteDisk(volumeName) if delerr != nil {

We should tag EBS volumes created by k8s with our tags, just like we (should) tag everything else we create/own.
Is there any documentation yet on automatic EBS volume creation?
from a user perspective, they are not AWS specific. That said, I don't know that we have clear user-facing docs. Here's an example though, from my (very) WIP Ubernetes-Lite docs: ... The PV is also tagged with the zone & region is is in (us-central1-a): Create a pod that uses the PVC:
It doesn't seem to cleanup the volume when the PV is deleted... probably intended behavior. Any way to tell it to automatically delete the volume on PV delete?
- I'm not sure about foo - hopefully that'll be resolved with docs. I think the failure to delete the volume is actually pretty important. Do you want to open an issue? It's not AWS specific (and this particular issue was auto-closed when the PR merged)
yeah I will open an issue later today. Issue seems to still be happening - the PV is never deleted. On Sat, Mar 12, 2016, 6:24 AM Justin Santa Barbara wrote:


Question: "//test/e2e/framework/resource:go_default_library", "//test/e2e/framework/service:go_default_library", "//test/e2e/framework/skipper:go_default_library", <ins> "//test/e2e/framework/testfiles:go_default_library", </ins> "//test/utils:go_default_library", "//test/utils/image:go_default_library", "//vendor/github.com/onsi/ginkgo:go_default_library",

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /cc /priority critical-urgent /milestone v1.19
Appears to have started failing after , which seems unrelated. /cc
according to the build log , the test timed out because it could not find gpu on any nodes. Is it probably a test-infra issue?
I think you are right, I don't see anything changing in the config so must be underlying infra issue /cc
yes, the issue is that the nodes do not have the GPU required to run the test The same infra commit () was used for the last passing job () and the first failing job ()
not sure who owns the node setup for the gpu tests, but I would guess
test-infra-maintainers owns e.g. prow, I don't think anything there relates to the GPUs the infra commit (test-infra commit) not changing suggests perhaps a change to the scripts / test harness in this repo?
the delta between the passing and failing runs only changed an ingress e2e test ()
The image is used in when installing the Nvidia driver. Given that ([1]) is lastly updated on July 17, 2020 at 12:22:11 AM UTC-7 (latest tag points to v2.0.0), does it seem this version is incompatible? BTW: the latest successful run pulled the image with SHA: 8d86a652759f80595cafed7d3dcde3dc53f57f9bc1e33b27bc3cfa7afea8d483 ([2]), maybe we can try that SHA specifically? [1]: [2]:
Forgot to mention, the shows that the deamonset was pending:
yes, and the log also shows that the pod was pending because the init container failed to complete could we change the image to  whose digest is exactly .


Question: // acceptedNames are the names that are actually being used to serve discovery. // They may be different than the names in spec. <ins> // +optional </ins> optional CustomResourceDefinitionNames acceptedNames = 2; // storedVersions lists all versions of CustomResources that were ever persisted. Tracking these

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, the field on must be explicitly set to their zero-values due to the fact that they are not tagged . Ref: Why is this needed: If using a system that automatically compares the provided configuration to the current, it will produce a diff on the updated status after creation. In addition, making sure to explicitly set to zero-values is not the best user experience. I am happy to provide the fix if this change is approved! /cc /sig api-machinery /area custom-resources
marking them as seems ok, so that client-side validation doesn't complain if they are absent when submitting manifests, but they shouldn't need to be set to omitempty (and at this point, we likely wouldn't change the serialization) Those fields are changed by controllers after creation, so expecting the live API object to exactly match the status is not going to go well anyway.
Good point! I’ll open a fix for this. Yep! The specific system in question ignores fields that are not set at creation time by default when doing comparison, so if status fields are not included then it will not report a diff.
/assign
I opened a fix for this but CC'ed both you and Let me know if any additional effort is required. Thanks!


Question: // if we are going to issue an update, be sure we track every name we update so we know if we update it too often. currResult.updatedItems.Insert(fsu.flowSchema.Name) <del> patchBytes, err := makeFlowSchemaConditionPatch(fsu.condition) if err != nil { // should never happen because these conditions are created here and well formed panic(fmt.Sprintf("Failed to json.Marshall(%#+v): %s", fsu.condition, err.Error())) } </del> if klogV := klog.V(4); klogV.Enabled() { klogV.Infof("%s writing Condition %s to FlowSchema %s, which had ResourceVersion=%s, because its previous value was %s, diff: %s", cfgCtlr.name, fsu.condition, fsu.flowSchema.Name, fsu.flowSchema.ResourceVersion, fcfmt.Fmt(fsu.oldValue), cmp.Diff(fsu.oldValue, fsu.condition)) } <del> fsIfc := cfgCtlr.flowcontrolClient.FlowSchemas() patchOptions := metav1.PatchOptions{FieldManager: cfgCtlr.asFieldManager} _, err = fsIfc.Patch(context.TODO(), fsu.flowSchema.Name, apitypes.StrategicMergePatchType, patchBytes, patchOptions, "status") if err != nil { </del> <ins> if err := apply(cfgCtlr.flowcontrolClient.FlowSchemas(), fsu, cfgCtlr.asFieldManager); err != nil { </ins> if apierrors.IsNotFound(err) { // This object has been deleted.  A notification is coming // and nothing more needs to be done here.

Integration test timed out due to the following error The error appeared 748 times in the log. Looks like lots of churning happens during the apf configuration bootstrapping phase. Is there a race condition between when the type is being registered and the API calls? Stack trace: Integration job link: PR: I searched for this error in CI and this PR, looks like we only see this error in this PR so far. So I am hoping it has not introduced any flake. (maybe we need to let more time to pass to start seeing flakes) We should not see this error I saw it only in the PR mentioned.
/sig api-machinery cc
I see
The fact that it happens to a non-mandatory object implies that the object was received from the server, so the type must be in the server. The APF config-consuming controller does this:
Look for the output from this line:
Another PR with this failure:
the panic means the request scope FieldManager is nil that is initialized here: OpenAPIModels is set here: called from here: and openAPIModels is constructed here: it looks like it is possible for openapiconfig to be unset, which causes the openapi models to be nil, which causes apply to panic. I don't know if it is only the integration tests that are subject to that configuration. cc /priority important-soon (since this is now being exercised in bring-up of integration test servers)
if APF is creating/bootstrapping the objects without using apply, and then updating status using apply, that would explain how the object exists but apply fails
I suspect that APF is not successfully updating status at all in the integration tests with this server configuration
do you want me to revert , while we work on a solution?
I don't know how bad the test impact is. Work with / to determine if we should revert or fix forward
: I do not follow why you talked about server-side stuff then suggested the problem is in the client (a controller in the kube-apiserver) not supplying a fieldManager. The client does supply the fieldManager, for both creates and updates. See the following lines:
Different field manager. is set server-side in the endpoint handler based on the server configuration.
So if the problem is server-side, why did you suggest the problem is in the client?

Regarding the client, I looked in and found this line: actually several of the same form, all with the same fieldManager.
I don't think I did suggest that :) all of the setup and calls in are server-side
: please help me understand . That is about the client side. Are you saying that a client that creates an object using and later updates the object using --- with a non-empty fieldManager in both --- will run into this server-side scenario? If so, is that because this client behavior is wrong or because there is a bug server side?
That was in response to , trying to figure out how the object was created but could not be updated. Yes, when the server is not configured to enable openapi and therefore does not have apply enabled. Because the server is misconfigured (which, from what I can tell, is limited to some of the server instances started for integration tests)
Looking at and it is not obvious to me whether or how openapi could be disabled. Would that be by turning off the feature flag? FYI, the inclusion of "V3" in the flag makes me unsure what the server does about other versions. : I think you are saying that it is possible to disable openapi and that this in turn disables server-side apply. That seems kind of surprising to me. Does this mean that it is simply incorrect to run a client that does server-side apply with a server that has openapi disabled? Is this intended? Is it documented anywhere?
It's not intended as an externalized configuration option. integration tests have the ability to fiddle with many of the config struct options in the server that a kube-apiserver binary invocation could not. They intentionally use this to do things like inject specialized authorizers / authenticators for testing purposes, but also can unintentionally result in a test server that doesn't match a real API server invocation. I think this is an integration test server configuration problem, which should be fixed.
is correct that we have a misconfiguration in the integration tests and SSA is only enabled on a subset of integration tests. I'd like to understand the test failure conditions a bit better though. The can be consistently reproduced on a non SSA targeted test. (Eg: selected from one of the test failures). However, I'm seeing the panic message while the test is still green and passes locally.

it's a bit bizarre what contributes to the flakiness of the test because the FieldManager panic can be reproduced 100% of the time.
The panic occurs when APF is updating status (which it just switched to do using SSA). The APF status update failure does not block the API requests the test cares about, so the test passes. I could see extra CPU usage contributing to flakiness.
Thanks for the explanation. Created to fix the integration test configurations.
/triage accepted
The integration test job started timing out dramatically last night (50% of runs) once merged. Since that was the only change made to integration tests recently, it seems related. I think we should revert (and therefore ) until the test impact is understood
revert of -
did this get fully reverted? we're still seeing panics in integration tests over this:
bumping priority and milestoning until we know we're not currently triggering panics
looks like it got reintroduced in , revert open in
For integration tests, maybe we can cheat and cache the OpenAPI data, so that it's not recomputed for each test?
:-/ maybe... but I really don't want to have non-real init paths in integration if at all possible
I wish we could go a little further in lazily initializing openapi (especially v2), so let's see what we can do here (after code-freeze probably)
if SSA is actively using the openapi, it being lazy won't help, right?
does the revert mean that we can remove milestone v1.24 from this issue?
Yes, cleared
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Taking a fresh look at this PR and wondering if it has been mostly fixed already? From comments my understanding is that this panic occurred due to two conditions: running as part of the integration test suite is asked to service an SSA request while are nil. The apiservice has logic to panic in this case. From the code path commented is nil when the server is created with is nil. APF worker (running alongside an integration test) issues an SSA request to the test apiserver. Since the apiserver is misconfigured it panics. This panic could happen sporadically because sometimes the tests finish before the APF worker is able to issue the request. As a brief check to see if this might be fixed, I ran the test mentioned was failing from SSA. I logged out the 's created and found that they all had managed fields. I saw this as an indication SSA was now functioning properly. Indeed, to start the test server the function calls . This function has a code path which unconditionally leads to the apiserver's being set. calls before running the returned apiserver: calls before using the returned config to create the returned apiserver: uses which sets : Thus, any test using should no longer be affected by this panic. The question then becomes: do all integration tests start their servers in this way? It's hard to say. I found a PR (, ) which refactors many occurrences of the old stanza: Into something using the new method: ` But it appears is still being used in a few integration tests: Should these occurrences be refactored to use too?
There was a ton of changes I did around that. But the last PR (already removing the old is already out for review): - FYI Thinking about it now, I guess I agree that the problem might have been 10 different way kube-apiserver was set up through different tests.

I just don't know why it was causing flakes, not failing consistently...
There were definitely test setups that were not enabling openapi in the recent past. Two possibilities for flakes: Enabling openapi in all integration test setups was expensive and caused spurious timeouts The excessive numbers of background panics handled and logged in integration tests when the API server tried to handle SSA requests bogged down the CI machine (I think something like 40,000 panics with full stack traces each were getting logged)
i don't think that's the case. With the recent changes that I was doing as part of I think OpenAPI is already enabled for all tests having kube-apiserver (almost all) and that seems to work. But you remainded me that back then we also had an issue that we were running many more tests simulanuously, which could have been contributing to this problem significantly. The second reason is definitely possible contributor too. I guess my concern was that if some requests were panicing due to lack of OpenAPI, this means that these aren't really needed. But that actually might make sense - the fact that we weren't able to upgrade status of PriorityLevels/FlowSchemas shouldn't prevent non P&F-tests from succeeding, and that worked in P&F tests...
I think the panics were happening in async controllers, and no functionality blocked by the controllers intersected with the surface being tested, it just (maybe) caused log/perf issues
I believe has resolved the panic issue, I was able to verify with which changes the priority & fairness controller to use server side apply. Before merged, I analyzed the integration log: Integration test log file from an unrelated pr does not contain any trace of the panic as expected: On the other hand, the log file from integration job for (before merged) had plenty of these panics even though the integration job turned green: I wanted to check which tests produced these panics, so I did a breakdown by package Then I narrowed it down to the individual test(s): Before merged, the above tests used , and either is or is which results in to be nil. But a quick code search revealed that many more integration tests used , so why were the other tests not affected by the panic?

I inspected the log of a test that was not affected by the panic, and grepped for , and found the following: In the above case, the apf controller had not advanced far enough, the controller was active for about , may be these tests finish quickly and signals the server to shutdown before the apf controller can send an APPLY command. On the other hand, log from a test that was affected by the panic shows that the apf controller was active for more than , and has lot more activities: I am assuming that integration tests have at least log level 4. Also, the size of the log file differed significantly, without panics around , and with panics over and changed all these tests to use and now integration log for shows no panic.
Great! (This also proves why eliminating gazilion different things of initializing stuff in favor of one makes sense). - FYI ^^ - let's merge the PR then - I just one question there, so let's wait for Jordan to respond it.
Can we just force the openapi/fieldmanager to be set at start-time now that SSA is GA, it should basically always be present, rather than fail when SSA is unavailable (that's probably wrong by the way). If that triggers any test flakiness, then we would solve these. We can wait for to be solved since it will almost certainly have a huge impact on this.
Has this been done Jeff?


Question: } func (s *ServiceController) createLoadBalancer(service *v1.Service) (*v1.LoadBalancerStatus, error) { <del> nodes, err := s.nodeLister.List(labels.Everything()) </del> <ins> nodes, err := s.nodeLister.ListWithPredicate(getNodeConditionPredicate()) </ins> if err != nil { return nil, err } <del> lbNodes := []*v1.Node{} for ix := range nodes { if includeNodeFromNodeList(nodes[ix]) { lbNodes = append(lbNodes, nodes[ix]) } } </del> // - Only one protocol supported per service // - Not all cloud providers support all protocols and the next step is expected to return //   an error for unsupported protocols <del> return s.balancer.EnsureLoadBalancer(s.clusterName, service, lbNodes) </del> <ins> return s.balancer.EnsureLoadBalancer(s.clusterName, service, nodes) </ins> } // ListKeys implements the interface required by DeltaFIFO to list the keys we

For reasons that are not entirely clear to me, and as spotted by in , there are two functions for selecting nodes in includeNodeFromNodeList & getNodeConditionPredicate. I think includeNodeFromNodeList should use the logic in getNodeConditionPredicate
/sig network


Question: // The number of old ReplicaSets to retain to allow rollback. // This is a pointer to distinguish between explicit zero and not specified. <del> // Defaults to 2. </del> <ins> // Defaults to 10. </ins> // +optional RevisionHistoryLimit *int32 `json:"revisionHistoryLimit,omitempty" protobuf:"varint,6,opt,name=revisionHistoryLimit"`

All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps


Question: // UpdateTransportConfig updates the transport.Config to use credentials // returned by the plugin. func (a *Authenticator) UpdateTransportConfig(c *transport.Config) error { <ins> // If a bearer token is present in the request - avoid the GetCert callback when // setting up the transport, as that triggers the exec action if the server is // also configured to allow client certificates for authentication. For requests // like "kubectl get --token (token) pods" we should assume the intention is to // use the provided token for authentication. if c.HasTokenAuth() { return nil } </ins> c.Wrap(func(rt http.RoundTripper) http.RoundTripper { return &roundTripper{a, rt} })

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With the exec credentials plugin configured in my kubeconf, and a stale/expired token present in the same file issuing e.g. triggers the authentication flow as handled by the plugin - once the flow is returned to kubectl (by printing the ExecCredential object) the new token is stored to the kubeconfig but not immediately used to resolve the request, leading to: Immediately issuing again works as kubectl now seem to use the credentials (token) stored following the first request. What you expected to happen: That kubectl would read the new token as fetched by the exec credential plugin and use it already for the first command issued. How to reproduce it (as minimally and precisely as possible): the kubernetes API server with the OIDC authenticator enabled. kubectl to use an exec credential plugin printing the token received out of band. that the token was indeed written to the kubeconf file. that the first kubectl command failed. that the next identical command succeeds. Anything else we need to know?: This works as expected when there is no token previously stored in the kubeconf - the behavior is only triggered when there is a token stored but it expired, or for whatever other reason not accepted by the API server. Environment: Kubernetes version (use ): 1.14, also tried with 1.17 Cloud provider or hardware configuration: OS (e.g: ): Mac OS for kubectl client Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig auth /sig cli
could you provide more details around how is configured. For example, it is unclear to me if you are using the built in oidc plugin or a custom exec plugin. I think you are using a custom exec plugin but its also trying to write tokens to the file (which it should not do)? /triage needs-information
Thanks for looking into it . You are indeed correct in your assumption that I'm using a which stores the token in the after retrieval. I'd love to learn why this should not be done, but the problem remains also without persisting the token after retrieval.

Basically the steps to reproduce could be reduced to these: containing an expired token. same configuring an exec plugin which retrieves a valid token out of band and echoes it. Expected behavior: The valid token retrieved is used for the subsequent authentication. Possibly expected behavior: The exec plugin flow not started (or exited with a warning) due to an expired token being present in the . Current behavior seems to acknowledge the invalid token and hence tries to retrieve a new, valid one - which it then discards as soon as it arrives.
Seems we can reduce it even further by removing the token altogether from the file and instead provide it directly. So the only step needed to reproduce is essentially this: configuring an exec plugin which retrieves a valid token out of band and echoes it as part of an . Now any will trigger the exec plugin to retrieve a valid token, but will still not use it for subsequent authentication.
This one is caused by config merging with clientcmd overrides: and exec.Auth not being in this switch:
Interesting, thanks Will add it to the switch along with some tests and if everything works out well I'll submit a PR next week.
Having spent some time on this in the debugger, it seems somewhat more intricate than I first thought. Turns out the exec plugin is executed whether the presented is invalid or valid - the exec action is actually called before the first call is even dispatched. This due to the exec credential plugin possibly being needed to setup the client certificate requested for the transport. Once the exec plugin's is called it assign the authenticators function to : ..which is then called when setting up the transport, thus triggering the exec plugin regardless of a token being present or not: Since the exec plugin has no way of knowing whether a token or a client certificate will be returned by the exec action, this kind of makes sense. It however defeats letting any present to override the exec plugin, which clearly seems to be the intent: Possible options here that I can think of: (and fix) behavior of letting override the exec plugin. This looks doable by adding a check for token auth early in the 's method body: But might be there are consequences to this I'm not aware of. the behavior of letting override the exec plugin being called.

third, more complex option, would be to keep the token override but dispatch to the exec credential plugin should the response from the first call with the token come back with a 401. This was what I originally believed was the intent as my first tests always included an invalid token. While this behavior seems nice I'm more and more thinking it may violate the principle of least surprise. Either you present a token and use that for auth or you let the exec plugin fetch the credentials for you. What do you guys think?
As a colleague just made me aware of, certain tools like kubetail seem to have kubectl call the exec plugin so many times that eventually the threshold of 1000 is reached and warn logs (exec:201) start spewing - even though no certificate (and hence no rotation) should be involved. This has me more convinced that the first option is the right one, skipping the client TLS ceremonies should a token be present for authentication.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: csiAttacher := getCsiAttacherFromVolumeAttacher(attacher, testWatchTimeout) pluginDir := csiAttacher.plugin.host.GetPluginDir(plug.GetPluginName()) testCases := []struct { <del> testName          string pvName            string expectedMountPath string shouldFail        bool </del> <ins> testName           string pvName             string volumeId           string skipPVCSISource    bool // The test clears PV.Spec.CSI shouldFail         bool addVolSource       bool // The test adds a Volume.VolumeSource.CSI. removeVolumeHandle bool // The test force removes CSI volume handle. </ins> }{ { <del> testName:          "normal test", pvName:            "test-pv1", expectedMountPath: pluginDir + "/pv/test-pv1/globalmount", </del> <ins> testName: "success test", pvName:   "test-pv1", volumeId: "test-vol1", }, { testName:        "fail test, failed to create device mount path due to missing CSI source", pvName:          "test-pv1", volumeId:        "test-vol1", skipPVCSISource: true, shouldFail:      true, }, { testName:     "fail test, failed to create device mount path, CSIVolumeSource found", pvName:       "test-pv1", volumeId:     "test-vol1", addVolSource: true, shouldFail:   true, </ins> }, { <del> testName:          "no pv name", pvName:            "", expectedMountPath: pluginDir + "/pv/test-pv1/globalmount", shouldFail:        true, </del> <ins> testName:           "fail test, failed to create device mount path, missing CSI volume handle", pvName:             "test-pv1", volumeId:           "test-vol1", shouldFail:         true, removeVolumeHandle: true, </ins> }, }

When two CSI PVs point to the same volume in the storage backend, kubelet can't start pods that use these two PVs on the same node. Both PVs have the same VolumeHandle and kubelet is smart enough to recognize that they're the same volume (because they have the same unique volume ID). It calls only once, however, the staging path contains name of one of the PVs, say PV1: One of the CSI volume calls then fails, because it uses PV2 to compute the staging path and that path does not exist. Both pods should just start. Using CSI HostPath volume, PV1 (dynamically provisioned for simplicity): PV2 (basically edited PV1): PVC1, PVC2, Pod1 and Pod2 are trivial, nothing special there. <detailsToday's master, </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority backlog
I suggest that if two volumes have the same unique volume ID (computed from CSI ), they must have also the same global mount path. Kubelet already has code to call NodeStage once and bind-mount it to different pods, regardless how many different PVs they use. These are several issues changing the global mount path: uses PV parsed from the global path to get the PV and to get enough info to call NodeUnstage: However, this parsing is used only when the volume plugin is not able to find/parse its json it has saved during MountDevice: Can we safely assume that the json is always there? the global path in a running cluster is tricky. We can require nodes are drained when kubelet with new global mount path is installed. This way, no CSI volume is mounted to a global path when the new kubelet starts. Minor cluster upgrades (1.23 -1.24) are , also documented in . Or we could fix kubelet to use the old global path when it already exists. This looks fragile to me.

staging dir needs to be unique enough not to conflict with other CSI volumes. It should be based on the driver name + volume handle, but that could be quite long. Can we compute a hash and hope it won't conflict?
I think in-tree driver can handle this case correctly because global mountpath is using unique volume id instead of pv name. With csi-migration, does it break backward compatibility?
We require users to drain nodes before enabling migration or when updating to a Kubernetes version that have migration enabled by default. All volumes should be therefore unmounted.
probably yes this breaks backward compatibility, but since only migration for PD is being done maybe it's more rare? We saw a customer case for this situation with NFS shares using the csi filestore driver. For PD I think it would only come up when trying to ROX a PD volume across namespaces, I'm not sure if that's common.
Capturing some thoughts here in line to understand your we use a staging path like , the kubelet can locate the both the driver name and the volumeId, needed during UnmountDevice call. e.g for pd csi driver which is the sha256{projects/mattcary-saikatroyc-test/zones/us-central1-c/disks/pvc--f97d-43cd-a5f1-}. This means the volume Id needs to persisted (we already do that in today ), so that the volume Id can be retrieved for UnmountDevice call. we decide to make the change at a minor boundary, then a node drain mandate would unstage any volumes before the kubelet is upgraded. this ensures any staged path for csi volume is using the new format 2 If we do not want to enforce a node drain, kubelet can first lookup json data (the reasoning is if this file does not exist, we will not be able to retrieve the volume handle from the new handle anyway), else fallback to try the old format of staging path - . TBD: Need to check mount reference logic. sha(volumeId) is expected to be unique in the cluster, if I understand. (based on the assumption any storage resource provisioned by a given driver is expected to be uniquely identifiable)
/assign
yes, your summary is correct. We persist a json, please make sure it contains anything that NodeUnstage may need.

It seems to me that people prefer option 2.2. We already do to uniquely identify VolumeAttachment. So far nobody complained about any conflict.


Question: } ]' spec: <ins> affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpress: - key: app operator: In values: - cockroachdb topologyKey: kubernetes.io/hostname </ins> containers: - name: cockroachdb # Runs the master branch. Not recommended for production, but since

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: Check , it is still using , we should move it to . What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
/assign
probably will want to move the init container also to the PodSpec
I think the init container still depend on annotations , can I specify in ? If so, I want to file another issue to track it
FYI Init containers have graduated to GA and now appear as a field. The beta annotation value will still be respected and overrides the field value. (,
Seems like that the doc should be updated. EDIT: NOTE: This feature is beta in Kubernetes 1.5. Once the feature exits beta, Init Containers will be specified in the PodSpec alongside the app containers array.
Thanks I filed an issue here
What about merge this first and then handle init-container in a separate PR?
Sounds good to me, thanks.


Question: "k8s.io/apiserver/pkg/authentication/user" genericapirequest "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/server/egressselector" <ins> utilflowcontrol "k8s.io/apiserver/pkg/util/flowcontrol" </ins> "k8s.io/component-base/metrics" "k8s.io/component-base/metrics/legacyregistry" apiregistration "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1"

On a cluster with many APIs provided by aggregated apiservers, with controllers managing those APIs, the number of occupied seats effectively has a floor of 1 per watch. If the combined max-inflight limit is reduced low enough, priority levels tend to become saturated by these long-running watch requests and other requests are starved. Watch requests consume a seat only during initialization. a PriorityLevelConfiguration and a corresponding FlowSchema to match watches against a resource served by an aggregated apiserver. some watch requests. the metric climb on the priority level per watch until all seats are exhausted. Kubernetes version (use ): (OpenShift 4.9) /sig api-machinery
cc
D'oh! I can see two obvious possible ways to cope: one apiserver not manage watch requests the are delegated to another apiserver; or a marker into the notification stream so that the non-origin apiservers can detect when initialization is done.
I would even say that for regular (non-long-running) requests, it's not obvious that they should be managed by P&F even. [I agree that for watch initialization it's particularly bad though.] - do we have a simple way to detect if the request will be served from our kube-apiserver or from some other aggregated apiserver?
There is not currently a way to know that ahead of time in the filters today. It require a list/watch, but it ought not be excessively expensive to make a 99% correct prediction, but the actual decision is made by the aggregated apiserver later in the request.
/triage accepted
: is there a way that we can plumb that knowledge forward to the filters?


Question: "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/yaml:go_default_library", "//vendor/k8s.io/apimachinery/pkg/watch:go_default_library", <ins> "//vendor/k8s.io/apiserver/pkg/util/feature:go_default_library", </ins> "//vendor/k8s.io/client-go/kubernetes:go_default_library", ] + select({ "@io_bazel_rules_go//go/platform:linux": [

Once the API for is finalized, add a node e2e test. /kind feature /assign /cc /sig node
We plan on adding this for 1.10


Question: err = f.WaitForPodRunning(pod.Name) framework.ExpectNoError(err, "Error waiting for pod to run %v", pod) <del> content, err = tk.ReadFileViaContainer(pod.Name, "test-container", testFilePath) framework.ExpectNoError(err, "Error reading file via container") gomega.Expect(content).NotTo(gomega.ContainSubstring(testContent)) </del> <ins> // for this to work, SELinux should be in enforcing mode, so let's check that isEnforced, err := tk.ReadFileViaContainer(pod.Name, "test-container", "/sys/fs/selinux/enforce") if err == nil && isEnforced == "1" { _, err = tk.ReadFileViaContainer(pod.Name, "test-container", testFilePath) framework.ExpectError(err, "expecting SELinux to not let the container with different MCS label to read the file") } </ins> }

Tests in are failing because they are faulty. A container can't read from a file if it has selinux permission denied for it. More context - /sig node /sig storage
something we can look into.
Also,
The tests expect that SELinux blocks a pod with label "s0:c2,c3" reading a file created by a pod with label "s0:c0,c1". Since the tests runs on GCI, where SELinux is not available, nothing blocks the read and the tests can't ever pass. The tests should check that the node has SELinux in Enforcing mode and skip the tests otherwise. Somewhere around these parts, where we write/read a file in the first "s0:c0,c1" pod: Is there an easy way how to test that in /proc or /sys in a non-privileged busybox container?
There are two problems. First one is obviously - the test can't run on non-selinux systems, but second one is - it is also broken on selinux enabled systems as per
To check whether SELinux is in enforced mode, one can check that read returns "1". I've implemented that in a particular case that relies on selinux being enforced: To check whether SELinux is enabled (I guess this is a prerequisite for all the test cases in ), one can use from (which of course should be run in a proper context). In case a check for a particular container is needed, the presence of directory should be enough. Having said that, I'm not sure how to properly wrap the above check(s) into (or , or whatever).


Question: wantPodStatus: v1.PodStatus{ Phase:   v1.PodFailed, Reason:  "Evicted", <del> Message: "The node was low on resource: memory. ", </del> <ins> Message: "The node was low on resource: memory. Threshold quantity: 2Gi, available: 1500Mi. ", </ins> }, }, }

when the pod is evicted due to low imagefs.available, the prompt is difficult to understand. it showed: But the container was evicted not by exceeding its request of 0 ephemeral-storage, but due to reaching the evictionHard of the node. The correct prompt message points out the reason for the pod eviction. don't print "which exceeds its request of 0.", because the container was evicted not by exceeding its request of 0. % oc label mcp worker custom-kubelet-eviction=small-pods evictionHard by creating a kubeletconfig % oc create -f custom-kubelet- custom-kubelet- several minutes, the mcp worker finish rolling out to the latest mc a pod % oc create -f hello-pod- hello-pod- a large file in container, make imagefs.available < 90Gi (the current available disk size is 111G ) command terminated with exit code 137 the node status the pod status % oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES hello-pod-2 0/1 ContainerStatusUnknown 1 7m58s 10.131.0.7 ip-10-0-213- <none<none% oc get pod hello-pod-2 -o yaml No response <details</details<detailsaws </details<details</details<details</details<details</details<details</details
/sig node
/assign
/assign
/triage accepted


Question: Code:    500, } return nil <del> } else if len(pdbs) == 1 { pdb := pdbs[0] pdbName = pdb.Name </del> <ins> } if len(pdbs) == 0 { return nil } pdb := &pdbs[0] pdbName = pdb.Name refresh := false err = retry.RetryOnConflict(EvictionsRetry, func() error { if refresh { pdb, err = r.podDisruptionBudgetClient.PodDisruptionBudgets(pod.Namespace).Get(pdbName, metav1.GetOptions{}) if err != nil { return err } } </ins> // Try to verify-and-decrement // If it was false already, or if it becomes false during the course of our retries, // raise an error marked as a 429. <del> if err := r.checkAndDecrement(pod.Namespace, pod.Name, pdb, dryrun.IsDryRun(deletionOptions.DryRun)); err != nil { </del> <ins> if err = r.checkAndDecrement(pod.Namespace, pod.Name, *pdb, dryrun.IsDryRun(deletionOptions.DryRun)); err != nil { refresh = true </ins> return err } <del> } return nil }) </del> <ins> return nil }) return err }() </ins> if err == wait.ErrWaitTimeout { err = errors.NewTimeoutError(fmt.Sprintf("couldn't update PodDisruptionBudget %q due to conflicts", pdbName), 10) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have about 8000 pdbs in our cluster. When we call several eviction requests at the same time, etcd will meet a performance spike. What you expected to happen: eviction request read pdbs from apiserver cache How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/sig scalability /cc throttling evictions at server-side may be a better fix i presume
We use eviction to auto migrate pods. If throttling eviction requests, it will make migration latency too long. I am not sure if list without rv by designed (get pdb consistently) or not (just forget to add rv).
What does eviction do (I can't remember from the top of my head)? Would you be able to quickly point me to the code?
Create create a eviction will list all pdbs, make sure eviction the pod won't break any pdb. If check passed, it will create the eviction, otherwise it will return 429 to reject.
Thanks - so this is what is problematic: So what you're saying is that you have 8000 PDBs in a single namespace? [because we only list pdbs from pod`s namespace]. I'm we should list from etcd, because otherwise we risk using old version of pdb ( are allowed to evict only 1 pod, and we just did it, but that wasn't yet propagated to apiserver cache and we would evict another pod). My personal guess would be that the problem may not really be number of PDB, but amount of concurrent requests touching the same pdb - because that would be resulting in conflicts and retrying them in: I'm not sure we can do much about it (at least I don't see any immediate low-hanging fruit).
Thanks a lot for your reply. How about retrying conflict update pdb, instead of re-listing all pdbs?

How about retrying conflict update pdb, instead of re-listing all pdbs? Hmm - I'm not sure I understand. Ohh, but I know what else we can do (maybe that's what you had on your mind). In case of conflict, instead of relisting all PDBs, we should simply GET the current version of that PDB (we already know that there is exactly one which we are supposed to use). That would reduce the cost of retries significantly. - would you be able to take a look into it and prepare a PR? I'm happy to review it.
- That's what I mean. I will take a look and send a PR in a few days.
- great! Please ping me on slack if I won't be responding to it.
alternatively, will pagination probably help ease the pressure? it'll skip the cache tho but relaxes the "spike"..
Does pagination mean range key prefix with limit?


Question: # mTLS should only be enabled for etcd server but not etcd-events. if $1 suffix is empty, it's etcd server. if [[ -z "${suffix}" && -n "${ETCD_APISERVER_CA_KEY:-}" && -n "${ETCD_APISERVER_CA_CERT:-}" && -n "${ETCD_APISERVER_SERVER_KEY:-}" && -n "${ETCD_APISERVER_SERVER_CERT:-}" && -n "${ETCD_APISERVER_CLIENT_KEY:-}" && -n "${ETCD_APISERVER_CLIENT_CERT:-}" ]]; then etcd_apiserver_creds=" --client-cert-auth --trusted-ca-file ${ETCD_APISERVER_CA_CERT_PATH} --cert-file ${ETCD_APISERVER_SERVER_CERT_PATH} --key-file ${ETCD_APISERVER_SERVER_KEY_PATH} " <ins> etcdctl_certs="--cacert ${ETCD_APISERVER_CA_CERT_PATH} --cert ${ETCD_APISERVER_CLIENT_CERT_PATH} --key ${ETCD_APISERVER_CLIENT_KEY_PATH}" </ins> etcd_apiserver_protocol="https" <del> etcd_livenessprobe_port="2382" etcd_extra_args+=" --listen-metrics-urls=http://${ETCD_LISTEN_CLIENT_IP:-127.0.0.1}:${etcd_livenessprobe_port} " </del> <ins> etcd_listen_metrics_port="2382" etcd_extra_args+=" --listen-metrics-urls=http://${ETCD_LISTEN_CLIENT_IP:-127.0.0.1}:${etcd_listen_metrics_port} " </ins> fi if [[ -n "${ETCD_PROGRESS_NOTIFY_INTERVAL:-}" ]]; then

Currently, etcd livenessProbe is defined here: This means that it proves /health endpoint 3 times (default failureThreshold value) and if all of them fails, it restarts etcd server. /health endpoint returns "unhealthy" if one of the conditions is met (): there is an active Alarm (there are two kinds of alarm: NOSPACE and CORRUPT) there is no raft leader the latency of a QGET request doesn't exceed 1s The problem is that in most of those cases, etcd's restart isn't a right behavior: if there is NOSPACE alarm, the restart will not free that space if there is no raft leader, it means that it cannot talk to other etcd members, so restart will not fix them if the request latency 1s, the etcd cluster is overloaded which is bad, but after restart we will get similar amount of requests (if not more), so restart isn't a right action e.g. in we see that event-etcd gets restarted by kubelet when it's overloaded. If we do not restart etcd at that point, we will likely simply finish work to be done and heal ourselves. I think we should change livenessProbe to detect cases where restart will actually help. /cc
I think that's not necessary true, that may potentially help sometimes. But I agree that's probably pretty rare. /sig api-machinery
Do we have an analysis of why the 3 healthchecks in row failed in Was it timeout, timeout, timeout or something else? Do you say that if etcd is overloaded then restarting it will help? Wouldn't it be even more overloaded and run into restart-loop because it will never become /healthy in time again?
Sorry, typo. I tried to say that in none of those cases restart will help (including 1s latency), but in fact I wrote something else. I changed initial post.
should fix etcd such that it does not declares as unhealthy as a result of overload (while its still making a progress). Ideally the not-healthchecking traffic should be throttled / load-shed, such that healthcheck stays responsive. this is implemented, I would consider giving higher timeout (e.g. 30sec), such that HC request have higher chance to reach beginning of the processing queue and get responded.

is a usual disjoint between 2 purpuses of healthchecking: a) autohealing -by restart -where we need to be very conservative about timeouts, as cost is high. b) measuring SLO & alerting -where we want to measure customer's happiness, and we cannot consider request waiting for 15s+ seconds as healthy/happy state. The lower tolerance more represents customers needs. In practical application we should optimize etcd healthcheck for a), while b) should be based on prometheous metrics.
This is basically the difference between "livez" and "readyz". "livez" is whether the app is running (for autohealing) "readyz" is whether it's useful from customer point of view (e.g. for SLO) In the ideal world, those should be two separate handlers.
In addition to doubling the timeout we can also consider halving the probing period (from default 10s to 5s) and doubling failure thresholds (from 3 failed probes in row to 6). It would decrease the probability of etcd being restarted in case of temporarily overloaded master.
There is no option to keep using "/health" endpoint and changing timeout to something higher, as this is hardcoded in etcd: Also I think we should stop restarting etcd on NOSPACE or CORRUPT alarms. This way we will stop restarting on alarms and have ability to control timeout (and setting it to some higher value). Once etcd starts exposing some better endpoint that can be used for liveness probes, we will switch to that. WDYT?
/cc /triage accepted


Question: if nanoSeconds <= 0 { return nil, fmt.Errorf("zero or negative interval (%v - %v)", newStats.Timestamp, cachedStats.Timestamp) } <del> usageNanoCores := (newStats.UsageCoreNanoSeconds.Value - cachedStats.UsageCoreNanoSeconds.Value) * uint64(time.Second/time.Nanosecond) / uint64(nanoSeconds) </del> <ins> usageNanoCores := uint64(float64(newStats.UsageCoreNanoSeconds.Value-cachedStats.UsageCoreNanoSeconds.Value) / float64(nanoSeconds) * float64(time.Second/time.Nanosecond)) </ins> // Update cache with new value. usageToUpdate := usageNanoCores

What happened: CRI CPU metrics are incorrect when using a large number of cores, due to a overflow. What you expected to happen: CPU metrics should be correct for arbitrarily high numbers of cores. How to reproduce it (as minimally and precisely as possible): Deploy a pod with 20 CPU units allocated, and fetch the CRI metrics. The metrics will be incorrect. Environment: Kubernetes version (use ): 1.14.4 CRI: CRI-O cc cc /sig node /sig scalability
How did you get the cri cpu metrics?
kubelet /stats/summary API. For "large" pods, we confirmed that cri-o/cgroupfs calculations were being generated correctly and CPU data overflows at the kubelet CRI stats provider layer. Overflows of CPU data in /stats/summary results in metrics-server having bad data which further propagates to the HPA scaling Pods incorrectly.


Question: # Accessing the Cluster  ## Using the Kubernetes proxy to access the cluster   * [Using the Kubernetes proxy](#proxy) * [Requesting redirects](#redirect) ## Using the Kubernetes proxy to access the cluster  Information about the cluster can be accessed by using a proxy URL and the cluster authentication keys.  For example, for a cluster that has cluster-level logging enabled (using Elasticsearch), you can retrieve information about the Elasticsearch logging on that cluster.   For example, if a cluster has cluster-level logging enabled (using Elasticsearch), you can retrieve information about the Elasticsearch logging on that cluster through a proxy URL. ### Retrieving the authentcation keys and proxy URLs Use `kubectl` commands to retrieve the access information.   To retrieve the configuration details about your cluster, including the address of the Kubernetes master cluster and the cluster's authentication keys (username, password, and access token), run the following command:   To retrieve the authentication keys for your clusters, run the following command:  ``` $ kubectl config view  { clusters: - cluster: certificate-authority-data: REDACTED server: https://104.197.5.247 name: kubernetes_logging contexts: - context: cluster: kubernetes_logging user: kubernetes_logging name: kubernetes_logging current-context: kubernetes_logging kind: Config preferences: {}   ...  users: - name: kubernetes_logging user:

is out of date. Please update for the current location/contents of the file and v1beta3. Once is submitted we should also update the examples to show to to curl using bearer tokens instead of basic auth credentials.
We're working on scrubbing the docs and examples of references to v1beta1 () prior to deleting that API, so it would be good to have this in soon.
: let me know if you want to punt this back to me -- now that I have some clue again about how to access an cluster (assuming I can create a cluster!).
Punting back to you. Getting the auth credentials is accomplished via But I think you have a better idea how to curl with token auth than I do.


Question: } function kube-up { <del> echo "Starting cluster using os distro: ${KUBE_OS_DISTRIBUTION}" >&2 </del> <ins> echo "Starting cluster using os distro: ${OS_DISTRIBUTION}" >&2 </ins> get-tokens

If e.g. coreos doesn't work with kube-up, we should point users to an installation method that works. Also consider offering hints like "vivid no longer gets security updates, consider using wily or official"


Question: type containerManagerStub struct { shouldResetExtendedResourceCapacity bool <ins> extendedPluginResources             v1.ResourceList </ins> } var _ ContainerManager = &containerManagerStub{}

A kubernetes cluster is normally constructed from a variety of machine templates. Different instance groups may have different CPU, memory capacity. They may be equipped with specialized accelerators, network devices. I think we should have an extensible way for users to custom node configuration to simulate real production cluster states. Currently, nodelabel and nodetaints are supported. I am consider to accept a template with more configuration like resources so we won't grow option list in the future. Without this feature, it's hard for users to simulate heterogeneous cluster.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scalability


Question: // all at once. APIListChunking featuregate.Feature = "APIListChunking" <del> // owner: @MikeSpreitzer @yue9944882 // alpha: v1.18 // beta: v1.20 // stable: 1.29 // // Enables managing request concurrency with prioritization and fairness at each server. // The FeatureGate was introduced in release 1.15 but the feature // was not really implemented before 1.18. APIPriorityAndFairness featuregate.Feature = "APIPriorityAndFairness" </del> // owner: @ilackams // alpha: v1.7 // beta: v1.16

Priority and Fairness is going GA in 1.29, the feature has been marked stable and the feature gate has been locked to default. See In 1.31, we need to remove the feature gate.
/kind bug /sig api-machinery /cc
/assign
/triage accepted


Question: vagrant ssh node-1 ``` <del> If you are running more than one nodes, you can access the others by: </del> <ins> If you are running more than one node, you can access the others by: </ins> ```sh vagrant ssh node-2 vagrant ssh node-3 ``` <ins> Each node in the cluster installs the docker daemon and the kubelet. The master node instantiates the Kubernetes master components as pods on the machine. </ins> To view the service status and/or logs on the kubernetes-master: ```console <del> $ vagrant ssh master [vagrant@kubernetes-master ~] $ sudo systemctl status kube-apiserver [vagrant@kubernetes-master ~] $ sudo journalctl -r -u kube-apiserver </del> <ins> [vagrant@kubernetes-master ~] $ vagrant ssh master [vagrant@kubernetes-master ~] $ sudo su </ins> <del> [vagrant@kubernetes-master ~] $ sudo systemctl status kube-controller-manager [vagrant@kubernetes-master ~] $ sudo journalctl -r -u kube-controller-manager </del> <ins> [root@kubernetes-master ~] $ systemctl status kubelet [root@kubernetes-master ~] $ journalctl -ru kubelet </ins> <del> [vagrant@kubernetes-master ~] $ sudo systemctl status etcd [vagrant@kubernetes-master ~] $ sudo systemctl status nginx </del> <ins> [root@kubernetes-master ~] $ systemctl status docker [root@kubernetes-master ~] $ journalctl -ru docker [root@kubernetes-master ~] $ tail -f /var/log/kube-apiserver.log [root@kubernetes-master ~] $ tail -f /var/log/kube-controller-manager.log [root@kubernetes-master ~] $ tail -f /var/log/kube-scheduler.log </ins> ``` To view the services on any of the nodes: ```console <del> $ vagrant ssh node-1 [vagrant@kubernetes-node-1] $ sudo systemctl status docker [vagrant@kubernetes-node-1] $ sudo journalctl -r -u docker [vagrant@kubernetes-node-1] $ sudo systemctl status kubelet [vagrant@kubernetes-node-1] $ sudo journalctl -r -u kubelet </del> <ins> [vagrant@kubernetes-master ~] $ vagrant ssh node-1 [vagrant@kubernetes-master ~] $ sudo su [root@kubernetes-master ~] $ systemctl status kubelet [root@kubernetes-master ~] $ journalctl -ru kubelet [root@kubernetes-master ~] $ systemctl status docker [root@kubernetes-master ~] $ journalctl -ru docker </ins> ``` ### Interacting with your Kubernetes cluster with Vagrant.

I cannot find such a service when . Same thing applies for . My cluster runs fine. cc:
Ah, good catch... everything is run as a container now, will update the documentation on where to get the logs.


Question: srcs = ["healthz_test.go"], embed = [":go_default_library"], deps = [ <ins> "//staging/src/k8s.io/api/core/v1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", "//staging/src/k8s.io/apiserver/pkg/endpoints/metrics:go_default_library", "//staging/src/k8s.io/component-base/metrics/legacyregistry:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: for extension API server panics with: Apparently introduces this panic with . What you expected to happen: to do not panic. How to reproduce it (as minimally and precisely as possible): extension API server with - for example the endpoint that the apiserver panics with Anything else we need to know?: Environment: Kubernetes version (use ): / Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind regression
/sig api-machinery
/assign
cc
Oops - sorry for that. It seems it is basically this comment: that I blindly applied without e2e retesting (the problem seem to be lack of tests, which is a bit surprising because bugs in my previous implementations were caught by tests). ^^ BTW - this has been cherrypicked to previous releases: 16: 17: 18: so it's actually even more critical...
I can file the automated cherry-pick PRs once the PR against the master branch is merged. :)
Reopening - since we need to cherrypick it still.
I now opened the cherry-picks: 16 - 17 - 18 -
Fix and picks are merged


Question: for vmssName := range vmssNamesMap { vmss, err := ss.getVMSS(vmssName, azcache.CacheReadTypeDefault) <ins> if err != nil { return err } </ins> // When vmss is being deleted, CreateOrUpdate API would report "the vmss is being deleted" error. // Since it is being deleted, we shouldn't send more CreateOrUpdate requests for it.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kube-controller-manager panic when wrong Azure VMSS name is configured: This is because the error is not validated first in the following code path: Azure cloud provider should check whether err happens and whether vmss is nil here. What you expected to happen: Azure cloud provider should report errors on such case, so that customers could get the root raw error and then fix the configuration. How to reproduce it (as minimally and precisely as possible): Create Kuberenetes cluster with basic load balancer, configure a wrong VMSS name and then create/delete LoadBalancer typed services. Anything else we need to know?: The issue also happens on older versions, so the fixes need to cherry-pick on other branches. /area provider/azure /sig cloud-provider /milestone v1.20 Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign


Question: }, }, }, <ins> "image whitespace": { expectedError: "spec.containers[0].image", spec: api.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "abc", Namespace: "ns"}, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSClusterFirst, Containers:    []api.Container{{Name: "ctr", Image: " ", ImagePullPolicy: "IfNotPresent", TerminationMessagePolicy: "File"}}, }, }, }, "image leading and trailing whitespace": { expectedError: "spec.containers[0].image", spec: api.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "abc", Namespace: "ns"}, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSClusterFirst, Containers:    []api.Container{{Name: "ctr", Image: " something ", ImagePullPolicy: "IfNotPresent", TerminationMessagePolicy: "File"}}, }, }, }, </ins> "bad namespace": { expectedError: "metadata.namespace", spec: api.Pod{

Is this a BUG REPORT or FEATURE REQUEST? BUG Kubernetes version (use ): kubernetes 1.5+ Environment: n/a What happened: examining events that happened in my cluster, i see an event like the following: this surprised me, as it implies a container had an imageName=" " What you expected to happen: i expected a validation failure error if a [i].imageName had leading or trailing whitespace. How to reproduce it (as minimally and precisely as possible): kubectl create -f my-pod kubectl edit pod/my-pod -o json change image to " " save Anything else we need to know: nope
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
discovered when looking at odd event messages in production cluster.


Question: }) } } <ins> func assignDefaultStartTime(pods []*v1.Pod) { now := metav1.Now() for i := range pods { pod := pods[i] if pod.Status.StartTime == nil { pod.Status.StartTime = &now } } } </ins>

What would you like to be : TestSelectNodesForPreemption should assign pod.Status.StartTime for test Pod Why is this needed: We can see the following in the output of TestSelectNodesForPreemption: This is due to missing assignment to Status.StartTime
/sig scheduling
/assign