Question: return statefulSet, pollErr } <ins> type updateJobFunc func(*batch.Job) func UpdateJobWithRetries(c clientset.Interface, namespace, name string, applyUpdate updateJobFunc) (job *batch.Job, err error) { jobs := c.Batch().Jobs(namespace) var updateErr error pollErr := wait.PollImmediate(10*time.Millisecond, 1*time.Minute, func() (bool, error) { if job, err = jobs.Get(name); err != nil { return false, err } // Apply the update, then attempt to push it to the apiserver. applyUpdate(job) if job, err = jobs.Update(job); err == nil { Logf("Updating job %s", name) return true, nil } updateErr = err return false, nil }) if pollErr == wait.ErrWaitTimeout { pollErr = fmt.Errorf("couldn't apply the provided updated to job %q: %v", name, updateErr) } return job, pollErr } </ins> // NodeAddresses returns the first address of the given type of each node. func NodeAddresses(nodelist *api.NodeList, addrType api.NodeAddressType) []string { hosts := []string{}

https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite} Happened on a presubmit run in . Previous issues for this test:
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}


Question: if err != nil { Failf("Failed to list nodes: %v", err) } <ins> // previous tests may have cause failures of some nodes. Let's skip // 'Not Ready' nodes, just in case (there is no need to fail the test). filterNodes(nodes, func(node api.Node) bool { return isNodeReadySetAsExpected(&node, true) }) if len(nodes.Items) < 2 { Failf("Less than two nodes were found Ready.") } </ins> podNames := LaunchNetTestPodPerNode(f, nodes, svcname, "1.4")

I ran e2e tests against docker 1.7.0-rc2 release, found a bunch of issues. Here is one I think it belongs to kubernetes for sure. 1) ResizeNode test failed, the machine is removed from instance group: But kubectl get nodes still returns 3 ndoes: This might be caused by the new docker binary I am testing against here. This is not the issue I am filing here. 2) Now networking test is running later, I noticed that the test is putting a pod on each node: nettest-gwtc4 on node e2e-test-dawnchen-minion-5l11 I haven't checked networking code in detail yet, not sure the issue is test itself, or scheduler issue. cc/ since you recently changed this. cc/
The networking test doesn't use the scheduler, it manually fills in the host field.
Ok, then we should filter out NotReady nodes from testing; otherwise, the test itself has the dependencies on other tests.
Yes, that test is doing a questionable thing.
Agreed.
I implemented skipping not-ready nodes in . Is it enough to close this issue, or is there something else to be done here?
I think that should fix the issue. ?
status: waiting for merge of


Question: // The IP protocol for this port. Supports "TCP", "UDP", and "SCTP". // Default is TCP. <ins> // +default="TCP" </ins> // +optional optional string protocol = 2;

It looks like the openapi is not routed properly to set-up the apiservice so apply requests will fail because the fieldmanager is not set-up properly. /cc /sig api-machinery /priority important-soon /kind bug
/wg apply /cc
OpenAPIModels is set to nil for the kube-aggregator server config (which serves the APIServices API): This is done because the kube-aggregator has a special openapi handler that aggregates all the other server's openapi docs. The field controls (at least) two things: /openapi endpoint serving (which we want kube-aggregator to continue to customize) creation of the field manager (which we want kube-aggregator to behave consistently on) We need to decouple presence of OpenAPIConfig and construction of StaticOpenAPISpec (which is needed by the field manager) from construction/installation of OpenAPIVersionedService. That might be a little tricky to untangle, since construction of StaticOpenAPISpec includes paths/operations. The field manager only needs the gvk/schemas.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen This is not something we want closed by accident.
/assign
For reference:
Additional lines for reference: This is roughly how SSA consumes the openapi spec. openapi spec -openapi proto model -typeconverter -SSA field manager For built in types, openapi spec -openapi model openapi model -typeconverter A similar thing happens for CRDs As liggitt mentioned, SSA only requires a subset of the information in the openapi generation, and the openapi generation in the aggregator slightly different than the process for builtins and CRDs.
/unassign
/assign


Question: // settings are ignored for this test. We could patch the image definitions. o := utils.PatchCSIOptions{ OldDriverName:            h.driverInfo.Name, <del> NewDriverName:            h.driverInfo.Name + h.driverInfo.Framework.UniqueName, </del> <ins> NewDriverName:            GetUniqueDriverName(h), </ins> DriverContainerName:      "hostpath", ProvisionerContainerName: "csi-provisioner", NodeName:                 h.driverInfo.Config.ServerNodeName,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Probably caused by <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
https://k8s- /assign
I understand that moved crd creation logic to e2e test's k8s cluster deployment code. On the other hand, I tested it with manually deployed k8s cluster. So, in my case, I guess that in addition to enable and feature gates enabled, I need to create crd manually before tests, like below: I could make two of three SkipAttach tests pass in my environment by above way. For "non-attachable volume does not need VolumeAttachment" case, I'm still looking into it. (It doesn't seem to skip creating volumeAttachment, properly)
I found a typo in 's argument. While hostpath driver's connects driver name and unique name without "-", it connects them with "-". This makes "non-attachable" test fail, due to the mismatch between volume name in crd and provisioner name, as a result, creating volumeAttachment isn't skipped. I will create a PR to fix it. For above bug, is unrelated. It just changed the way in how to test in manually deployed k8s cluster as I mentioned in above comment. So, there should be no problem for the PR, at least for the CI testing if it works well in CI environments.
I can verify that the CRDs are not getting installed on the alpha cluster. Investigating why.
/reopen since I see the Alpha tests failing
/milestone v1.13 /priority important-soon /kind failing-tests
/kind failing-test
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is fully resolved. There are still failures on https://k8s- /reopen
Reopened this issue. <detailsIn response to : I don't think this is fully resolved. There are still failures on https://k8s- Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you help take a look at these failures? It's testing CSI filesystem volume mode. I think it is unrelated to the crd installation issue that Saad fixed.
I tested with AllAlpha flag and manually created csi crds and confirmed that filesystem volume mode test passed in my environment and also other tests passed there. If I delete csi crds, other tests also fail, so I agree that the issue that Saad fixed is not related. However, above also means that this issue can't be reproduced in my environment.
I1109 14:56:01.141] Nov 9 14:56:01.131: INFO: At 2018-11-09 14:50:54 +0000 UTC - event for security-context--e42e-11e8-9b95-: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc--e42e-11e8-880b-" : node "bootstrap-e2e-minion-group-3wtg" has no NodeID annotation
Hostpath also seems to be failing because of driver registration issues? Not sure why other alpha tests are passing if that's the case
This test is the only test that uses  to create pod with CSI driver volume and expects the result to be success. One of the pod parameters specified there might affect this failure.
Those 2 failing tests are newly introduced in and started to fail from the beginning. Is that PR blocking 1.13? If not, can we revert and fix later? I have recently set up optional presubmit tests for alpha features at , which will pick up those 2 tests. Currently this failure is blocking
apart from the 2 failing tests, I see which addressed with is also flaky. These failures are now blocking PRs as mentioned above. As we enter code slush today, its very critical to have the PRs get in sooner than later. Can we plz revert the feature and the tests until they are fully resolved and ready to go. Thanks
I prefer disabling the specific tests instead of reverting the entire pr. We will lose important signal for csi
That works for me.

Can you cc me on the PR that disables those tests? When you re-enable the tests, remember to try them out on . :)
Let me consider if we can just skip these test for CSI, instead of reverting the entire patch.
you mean just skipping them in Alpha cluster right?
I was thinking about skipping the below test with the combination of below drivers: [Test] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode[Feature:BlockVolume] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources [Driver] csi-hostpath Won't it enough? Or, do we need to skip it only with alpha cluster?
If we just disable the tests alone and not revert the feature, we are just masking the issue. Ideally if we are not able to fix the tests and reenable them the feature should be reverted from 1.13 as well. Are you also actively working on fixing the tests and this skip is just to unblock PRs in the meantime? if so can we timebox by when we can have the tests fixed and re-enabled / the feature reverted? My fear is we might forget the fact that its disabled since there's no active reporting on it.
Actually, there is no feature in either of culprit commits. It tests or intended to improve the way it is tested. Therefore, it must be a test issue or another issue which is found by the test , not an issue introduced by the commits themselves. In the code, there is a clear comment that the test is skipped due to this issue with the reference to this issue#, so this won't be forgot. If you still worry about this issue is not tracked, we might be able to keep this issue open.
IIUC the test that were failing (and are being reverted) were for the new CSI festures going to Alpha in 1.13, right? so even these 2 culprit commits do not contain feature work, it is testing some new code additions to 1.13 and now those will be untested until we re-enable these tests. Is there an ETA to re-enable them, preferably sometime early next week? Yes +1 to keeping this issue open to track the skipped tests.
The two failing tests themselves are not for new alpha features that are in v1.13.

Instead, they are just to test csi driver works well as it works in existing in-tree driver. (And failed because they might be affected by some other alpha features or some of specific configurations in gce.) In addition, ~50 tests for csi driver, so it would be better not to remove all of the tests by reverting the PR, just for the 2 failing tests. I think that just reverting the patch won't help improving quality of k8s. As for ETA to solve the issue, I found difficulty in debuging in my local environment. It seems that the two tests fail in gce environment w/ AllAlpha=true, however, as I've already mentioned, at least csi-hostpath tests in my local environment w/ AllAlpha=true doesn't fail.(And I don't have test environment for gce.) Of course, I will help solving the issue next week. However, I will need someone's help for testing combinations of alpha features in gce environment to narrow down which flag really affects this issue. At least I would like to know the results: w/ MountPropagation=true,CSIPersistentVolume=true,BlockVolume=true,CSIBlockVolume=true all alpha flags except CSIDriverRegistry and CSINodeInfo
/reopen Thanks for the detailed explanation. I think the revert is fine in this case then. I am keeping this issue open however to track and fix the failures in these 2 tests can any of you help answer w.r.t alpha features on gce environment. thanks
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can help debug when I'm back on Tuesday. In the meantime, I would suggest looking through the failing test logs to try to triage what's going on
Here's one failing run:
I looked into the csi-hostpath failure log. As seen in the for , where the pod which has a csi-hostpath volume is being attached is scheduled, mount volume fails with these errors (1). Related code to this log is (2), so the driver doesn't seem to be registered to . As for the registration code for csiDriver, it is called as (3).

Therefore, will be logged if the driver is triggered for registration. As seen (4), registration is only triggered in , which is different from the one the pod is scheduled. In addition, driver-registrar seems to be deployed only on the node where registration is triggered (5). Is it right behaviour, while registrar is deployed as daemonset? I guess that just this fact seems to cause an issue to make csi-hostpath driver work properly. (Actually, I only tested with 1 node environment, so this might only happen in multiple nodes environment.) <details- (1) (2) Result of grep to check whether registrar is deployed to each node [ for bootstrap-e2e-minion-group-50r7][ for bootstrap-e2e-minion-group-k781][ for bootstrap-e2e-minion-group-z5cg] </details
The hostpath plugin should be deployed as a DaemonSet, so it's strange that it's only getting registered on only one node.
/assign looking into this as well
could you link the PR for skipping [external-gcepd] tests into this issue when you do that as well
skipping gcePD-external is here:
Since gce-cos-alphafeatures is now passing, do we need to keep this issue open? Should it morph into a non-blocking issue for longterm work on a better test?
We have root caused 2 of the 3 failures, which were problems in the test and test conifg. Fixes will be out soon. Still triaging the 3rd failure, which could be a real bug in alpha clusters
I've root caused csi-hostpath failure issue. It just failed to set the node properly in test code side. I will make a PR to fix it.
Or should it be done in your PR too, to avoid modifying twice in the separate PR? Actually, the fix are as below: (Failing to set node before pod creation)
Created for csi-hostpath fix, to prepare for the case that we need a separate PR. We can discuss more on this there, if needed.
/unassign found possible root cause for GCE issue: CSI NodeID annotation is not present on the Node object, could be caused by failed retries on updating Node object.

(apparently when alpha features are open we shouldn't be looking at the Node object anyway except we seem to be, so that might also be a bug). has the most context for the fix /assign
The symptom of the remaining test failure is the test pod fails to create because it fails to attach with the error: . The test succeeds the first time it's run in a cluster, and always fails after that. Two issues caused this error message. the CSI external-attacher tries to fetch the object to get the NodeID, it does so through an informer. The first time the test is run, objects are created for the very first time, and the informer returns the correct object. In subsequent test runs, s from the first run is kept, and for some reason the informer fails to detect the object and returns empty. Making the API call directly works, I'll patch the external-provisioner with this to mitigate the issue, and this is sufficient to get the test to pass. The informer bug will need to be eventually fixed. fails to post the NodeID annotation sometimes, likely due to node status patch not being retried correctly. This is crucial for clusters without feature flag set, and this is the default for 1.13. Will add the appropriate retry mechanism.
Opened to track the informer issue
Are and potential PR for the only 2 fixes being actively worked on for this issue?
Only is needed. We have a workaround for
will you have a followup PR to enable the CSI alpha tests back? or is that already in flight?
also reenables the tests
OK with merged we should see the CSI alpha tests re-enable. to keep an eye out for CI signal
The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! /close
Closing this issue. <detailsIn response to : The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The tests are actually running (and passing) in the normal suite now, now that the raw block feature moved to beta.


Question: var _ = SIGDescribe("ReplicationController", func() { f := framework.NewDefaultFramework("replication-controller") <ins> var ns string var dc dynamic.Interface ginkgo.BeforeEach(func() { ns = f.Namespace.Name dc = f.DynamicClient }) </ins> /* Release : v1.9 Testname: Replication Controller, run basic image

According to this APIsnoop query, there are still some remaining ReplicationController endpoints which are untested. Create a ReplicationController with a static label Patch the ReplicationController with a new Label and updated data Patch the ReplicationController's status Patch/Scale the ReplicationController's Replica count to 2 Replicas Get the ReplicationController Replace the ReplicationController's Status List all ReplicationControllers in all Namespaces find the ReplicationControllers(1) ensure that the ReplicationController is found and is patched Delete Namespaced ReplicationControllers(1) via a Collection with a LabelSelector Discover useragents: See all endpoints hit from this test Display test coverage change If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
Deployment replaced ReplicationController a while back. It should be part of conformance. We need to check more than the data-aspect (beyond crud), we need to verify the data-plane. In this case we need to check the pods as well and not just the ReplicationController status. Moved to backlog by conformance subgroup.
PR:
/open /reopen We need this open to track the test promotion and flakyness. We can't write a promotion PR until the test has no flakes for 2 weeks.
Reopened this issue. <detailsIn response to : /open /reopen We can't write a promotion PR until the test has no flakes for 2 weeks. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/retitle Promote CoreV1ReplicationController lifecycle mock test - +7 coverage Need to verify two weeks on non-flakeyness.
/retitle Write CoreV1ReplicationController lifecycle test+promote - +7 coverage
Closing in favour of /close
Closing this issue. <detailsIn response to : Closing in favour of /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: e2eservice.WaitForServiceUpdatedWithFinalizer(cs, svc.Namespace, svc.Name, true) }) <del> ginkgo.It("should find a service from listing all namespaces", func() { </del> <ins> /* Release : v1.18 Testname: Find Kubernetes Service in default Namespace Description: List all Services in all Namespaces, response MUST include a Service named Kubernetes with the Namespace of default. */ framework.ConformanceIt("should find a service from listing all namespaces", func() { </ins> ginkgo.By("fetching services") svcs, _ := f.ClientSet.CoreV1().Services("").List(metav1.ListOptions{})

This issue proposes the writing of a new test for the API endpoint with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. There are no tests listing services, see the following: This mock test uses client-go's functions and hits the endpoints. This shows the endpoints hit by my approach: /sig architecture /area conformance


Question: "k8s.io/apimachinery/pkg/util/uuid" runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1" "k8s.io/kubernetes/test/e2e/framework" <del> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" </del> imageutils "k8s.io/kubernetes/test/utils/image" admissionapi "k8s.io/pod-security-admission/api"

Test: [sig-node] Restart [Serial] [Slow] [Disruptive] Kubelet should correctly account for terminated pods after restart I've also seen on /kind flake /sig node Looks like occasionally we have a second 50m CPU pod hanging around on the node, blocking enough pods getting scheduled for the test: I'm a bit puzzled because no pods are listed as currently running when the node dumps debug info. Could be a bug?
another case [sig-node] Container Manager Misc [Serial] Validate OOM score adjustments [NodeFeature:OOMScoreAdj] affected the Restart test case, see the kubelet log, the kubelet-container-manager guaranteede9d81584 pod was received almost simultaneously with the restart-test pod
Since at 0316 21:25:08, the container of pod guaranteede9d81584 has just been creating, but the test case [sig-node] Container Manager Misc [Serial] has already finished at 0316 21:25:06, so the test case check condition may be not strict
/triage accepted /assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if len(vpcs.Vpcs) != 1 { return nil, fmt.Errorf("Error querying VPC for ELB, got %d vpcs for %s", len(vpcs.Vpcs), c.vpcID) } <del> return vpcs.Vpcs[0].CidrBlock, nil </del> <ins> cidrBlocks := make([]string, 0, len(vpcs.Vpcs[0].CidrBlockAssociationSet)) for _, cidr := range vpcs.Vpcs[0].CidrBlockAssociationSet { cidrBlocks = append(cidrBlocks, aws.StringValue(cidr.CidrBlock)) } return cidrBlocks, nil </ins> } // abstraction for updating SG rules

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When an AWS NLB is created through the Service of type, the health check security group rule is only created with a VPC primary CIDR block, as seen here: If the VPC has multiple CIDR blocks associated, and the worker nodes are running on those corresponding subnets, the health check fails. What you expected to happen: Multiple health check security group rules are created, one for each VPC CIDR block, using both and (). How to reproduce it (as minimally and precisely as possible): a VPC with with additional subnets created on a non-primary CIDR. a Kubernetes cluster (e.g. EKS) on subnets associated with a non-primary CIDR. a Service of type with nodes are healthy due to the security group rule being created only allowing traffic from the primary CIDR. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Amazon Linux 2 Kernel (e.g. ): Install tools: N/A Others: N/A
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, i'll provide a fix next week


Question: newPod, ok := obj.(*api.Pod) // Check whether the object could be converted to single pod. if !ok { <del> err = fmt.Errorf("invalid pod: %#v", obj) return false, pod, err </del> <ins> return false, pod, fmt.Errorf("invalid pod: %#v", obj) </ins> } // Apply default values and validate the pod.

To read a pod manifest file from disk (or from the http source), kubelet execute the following code to Decode a on-disk v1 pod manifest to an internal pod object (the defaults are applied in this process). Validate the internal pod. Convert the internal pod back to a v1 pod. The downstream pod cache (which gets v1 pods from different sources) performs similar operations: Convert the v1 pod to an internal pod. Validate the internal pod. One of two conversion/validation should be removed, and some of them should be simplified.
/cc for suggestions on defaulting and conversion.
If no one is working on this, I will take it.


Question: klog.ErrorS(err, "Failed to parse a pod full name", "podFullName", podFullName) return false, err } <del> klog.V(2).InfoS("Deleting a mirror pod", "pod", klog.KRef(namespace, name), "podUID", uid) </del> <ins> var uidValue types.UID if uid != nil { uidValue = *uid } klog.V(2).InfoS("Deleting a mirror pod", "pod", klog.KRef(namespace, name), "podUID", uidValue) </ins> var GracePeriodSeconds int64 if err := mc.apiserverClient.CoreV1().Pods(namespace).Delete(context.TODO(), name, metav1.DeleteOptions{GracePeriodSeconds: &GracePeriodSeconds, Preconditions: &metav1.Preconditions{UID: uid}}); err != nil { // Unfortunately, there's no generic error for failing a precondition

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Consider the following log line: I expected the podUID to be printed rather than an address/pointer. Run a static pod on a non-standalone kubelet, watch logs for mirrorclient entries. Fix is this line: We probably want to check if the UID is nil, and if so, print what it points to. /sig node /area logging /help Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --#### What happened: /area logging /help Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: </details
/assign
/triage accepted


Question: glog.V(2).Infof("About to try and schedule pod %vn"+ "tknown minions: %vn"+ "tknown scheduled pods: %vn", <del> pod.Name, minionCache.Contains(), podCache.Contains()) </del> <ins> pod.Name, minionCache.ContainedIDs(), podCache.ContainedIDs()) </ins> return pod }, Error: factory.makeDefaultErrorFunc(&podBackoff, podQueue),

The client/cache interface has a method which is supposed to return a of the IDs in a store. This is a bit misleading and should probably be renamed to 'Contents' or 'ContainedIDs'.
Thoughts
I tend to agree. contains to me should always return a bool and take an input key. Sent from my iPhone
Yeah, I agree, both of the suggested names would be an improvement.


Question: for _, check := range checks { s.livezChecks = append(s.livezChecks, delayedHealthCheck(check, s.livezClock, delay)) } <del> return s.addReadyzChecks(checks...) </del> <ins> return nil </ins> } // addReadyzShutdownCheck is a convenience function for adding a readyz shutdown check, so // that we can register that the api-server is no longer ready while we attempt to gracefully // shutdown. func (s *GenericAPIServer) addReadyzShutdownCheck(stopCh <-chan struct{}) error { <del> return s.addReadyzChecks(shutdownCheck{stopCh}) </del> <ins> return s.AddReadyzChecks(shutdownCheck{stopCh}) </ins> } // installHealthz creates the healthz endpoint for this server

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When using GenericAPIServer you can only add new checks via function which adds them to all , , and endpoints. All other methods like are private. I would like to request those methods to be made public: Why is this needed: Developers of apiservers should be able to define their readiness probes add them to specific endpoint of their choosing. For example in case of Metrics Server we defined probes: Readiness - Metrics cache is full. As MS serves data only from cache, traffic will be only sent after cache is full. Liveness - Metric scraping loop is not deadlocked. It doesn't make sense to have those probes to both endpoints. Not ready metric cache should not cause MS being killed via liveness probe. Delayed scrape should not cause traffic not being served from cache due failing readiness probe. /assign
/sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig apimachinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/cc /triage accepted
I can see this. PR welcome.


Question: // Never cache this.  The access token is already // cached by the metadata service. &containerRegistryProvider{ <del> metadataProvider{Client: http.DefaultClient}, </del> <ins> metadataProvider{Client: &http.Client{Timeout: 10 * time.Second}}, </ins> }) }

http://pr-
Something holding the port already?
The previous masterless case just barely passed with a 12.5 second timeout (25 tries at 0.5 second interval) I think we just need to increase the timeout.
I randomly sampled 5 green builds and they all have this test (starting kubelet in masterful/less mode) finished in 3s or less. Judging for the successful masterless mode startup before the failure, kubelet spent the most time trying to register credential providers (a 10s gap in the log). The same roughly applies to the failed "masterful mode", based on where the log message stopped. From , the registering process basically tries to access the metadata url to see if the credential provider is enabled. I think this may just be the slow response from the metadata server.
I am wondering even if this issue might be related to a slow response from the metadata server, why it does not show "Registering credential provider: .dockercfg" always in failed logs?
If there is a possibility that ReadUrl in may take longer than expected, would it be preferred to use a Timeout (0) with the http client to not get stuck longer than expected?
I am unsure why it didn't show that, but even in the passing round we see a 10 second gap, which almost maxed out the timeout. I think adding a timeout is reasonable, but I am not sure what value is reasonable given that the previous request took 10s. do you have any idea?
Even I am not sure exactly what should be a reasonable value. My guess would be that if in normal cases, average response time is x seconds (perhaps a guestimate could be computed based on "normal" successful tests), then x could be a starting point, or otherwise another option is choosing something randomly (like 5s). And I think that if at some point, the server is responding in 10s, and even though the test passes, its not normal and something is going wrong.
uses a 10 second timeout, but the request will be retried. Given that we don't retry this request, a short timeout is worrisome. Maybe we should add both retry and timeout in this case.
SGTM.
I reopen this one since pr was reverted.

Noticed a similar hang in: The logs are a bit different from those observed in
if kubelet couldn't start up properly, the test itself should've timed out and logged "Timed out waiting for kubelet to answer at http://127.0.0.1:/healthz; tried 25 waiting 0.5 between each". I am not sure if this failure is related.
failure is not relevant to this issue. The most recent report of such flake was over a week ago. I think the cause is the slow metadata url response in GCE, and that should've been fixed by the recent rollout. I am going to close this issue. Feel free to reopen if there is another occurrence.


Question: able to start with Secret and ConfigMap volumes mounted into the container. release: v1.13 file: test/e2e/storage/empty_dir_wrapper.go <ins> - testname: PersistentVolumes(Claims), apply changes to a pv/pvc status codename: '[sig-storage] PersistentVolumes CSI Conformance should apply changes to a pv/pvc status [Conformance]' description: Creating PV and PVC MUST succeed. Listing PVs with a labelSelector MUST succeed. Listing PVCs in a namespace MUST succeed. Reading PVC status MUST succeed with a valid phase found. Reading PV status MUST succeed with a valid phase found. Patching the PVC status MUST succeed with its new condition found. Patching the PV status MUST succeed with the new reason/message found. Updating the PVC status MUST succeed with its new condition found. Updating the PV status MUST succeed with the new reason/message found. release: v1.29 file: test/e2e/storage/persistent_volumes.go </ins> - testname: PersistentVolumes(Claims), lifecycle codename: '[sig-storage] PersistentVolumes CSI Conformance should run through the lifecycle of a PV and a PVC [Conformance]'

[x] test approval issue : [x] test pr : [x] Flake [x] two weeks soak end date : 31 Oct 2023 [X] test promotion pr : [X] remove endpoint from According to following APIsnoop query, there are three PersistentVolume and three PersistentVolumeClaim endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 6 endpoints. The e2e logs for this test are listed below. This query shows the following PV and PVC endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 6 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted /cc
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: err = e2epod.WaitForPodSuccessInNamespace(f.ClientSet, podClient.Name, ns.Name) if err != nil { // Collect pod logs when we see a failure. <del> logs, logErr := e2epod.GetPodLogs(f.ClientSet, f.Namespace.Name, podClient.Name, fmt.Sprintf("%s-container", podClient.Name)) </del> <ins> logs, logErr := e2epod.GetPodLogs(f.ClientSet, f.Namespace.Name, podClient.Name, "client") </ins> if logErr != nil { framework.Failf("Error getting container logs: %s", logErr) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In tests, always fails when the tests fail, with the error: . This is the message indicating that APIServer responds 400 code. What you expected to happen: Container logs should be collected correctly if tests fail. How to reproduce it (as minimally and precisely as possible): a NetworkPolicy test to make it fail and run it, or run NetworkPolicy tests with a CNI that doesn't implement NetworkPolicy. the tests fail, all attempts to get container logs will fail like the following. Anything else we need to know?: Environment: Kubernetes version (use ): 16.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing /assign
/remove-sig testing /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: if !rc.desiredStateOfWorld.VolumeExists( attachedVolume.VolumeName, attachedVolume.NodeName) { // Volume exists in actual state of world but not desired <ins> // Mark desire to detach timeElapsed, err := rc.actualStateOfWorld.MarkDesireToDetach(attachedVolume.VolumeName, attachedVolume.NodeName) if err != nil { glog.Errorf("Unexpected error actualStateOfWorld.MarkDesireToDetach(): %v", err) } // Update Node Status to indicate volume is no longer safe to mount. err = rc.nodeStatusUpdater.UpdateNodeStatuses() if err != nil { // Skip detaching this volume if unable to update node status glog.Infof("UpdateNodeStatuses failed with: %v", err) continue } </ins> if !attachedVolume.MountedByNode { glog.V(5).Infof("Attempting to start DetachVolume for volume %q from node %q", attachedVolume.VolumeName, attachedVolume.NodeName) err := rc.attacherDetacher.DetachVolume(attachedVolume.AttachedVolume, rc.actualStateOfWorld)

introduced logic to make kubelet volume manager wait for attach detach controller to finish attach before it starts mount. The controller should also not start a detach operation until the node status has been updated removing the volume from the list of attached volumes. Otherwise if a pod is rapidly created and deleted on the same node, it could result in a mount starting immediately before a detach starts. Props to and his pet set tests for helping catch this.
Hmm, this just happened , just a lot less frequently. Haven't debugged, but it looks like the failed pod did reland on the same node: pet-0, jenkins-e2e-minion-group-4mn2 (). Reopening, will try to confirm when i have time, and close if it's not the same thing.
Just following up with an update, Saad and i are tag teaming this. He's uploading a pr that I'm going to reivew. We're tracking a couple of issues: need to update the "volumes in use" list before polling on the "volumes attached" list. Best option is to write a operation executor, so we don't hammer the apiserver. needs to get up to date node object from apiserver to determine volumes in use. The update that removes volumes from the "attached list" shoud return a fresh node object, we can just iterate the volumes its using (this must happen after popping from attached, though). leaky volumes in the node object (this is not about the physical volumes but about the size of the json blob that's the node object growing). The proposed fix is to use the desired state of the world to update the list of volumes in use. There's a last point I need to follow up on. We might leave the node object in an inconsistent state with the proposed fix, but it should be ok since nothing is checking the inconsistent fields. This will happen if we pop a volume from the attached list, but then find its in use in the updated node object. ping me when you're ready to go, of if you need help


Question: LogsForObject    polymorphichelpers.LogsForObjectFunc genericclioptions.IOStreams <ins> TailSpecified bool </ins> } func NewLogsOptions(streams genericclioptions.IOStreams, allContainers bool) *LogsOptions {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When attempting to get all log lines using only 10 lines of logs are returned What you expected to happen: All log lines should be returned How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: This was introduced in in an attempt to fix another issue where selectors would ignore tail values !=-1. I think the problem here is the logic cannot tell if the was set explicitly by the command line or is simply the initial default value. Environment: Kubernetes version (use ): 1.13.3 Cloud provider or hardware configuration: Kubeadm OS (e.g: ): 18.04.2 LTS (Bionic Beaver) Kernel (e.g. ): Linux ubuntu 4.15.0-45-generic -Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig cli
Maybe is not a bug. See the kubectl help. --tail=-1: Lines of recent log file to display. Defaults to -1 with no selector, showing all log lines otherwise 10, if a selector is provided. I think i mean a selector is provided the --tail should not use -1 ?
Thanks for you reply. There are some background in below tickets, --tail=-1 is one of the option. Please check.
The help indicates that the default is 10 if the selector is used. But if someone explicitly asks for -1 it should still work, at least in my opinion.
Encountering similar behaviour to this myself. Is there any chance whatever fixes get applied can also be applied to the branch? Since that also has strange behaviour around logs with label selectors and setting .

See
This ticket has two issues: First, It's said "TailLines must be greater than or equal to 0", however we can specify -1. kubectl logs -l app=test --tail=-3 error: TailLines must be greater than or equal to 0 kubectl logs -l app=test --tail=-1 : Wed Apr 10 06:51:34 UTC 2019 : Wed Apr 10 06:51:35 UTC 2019 : Wed Apr 10 06:51:36 UTC 2019 : Wed Apr 10 06:51:37 UTC 2019 : Wed Apr 10 06:51:38 UTC 2019 : Wed Apr 10 06:51:39 UTC 2019 : Wed Apr 10 06:51:40 UTC 2019 : Wed Apr 10 06:51:41 UTC 2019 : Wed Apr 10 06:51:42 UTC 2019 : Wed Apr 10 06:51:43 UTC 2019 Second, --tail=-1 is the legacy usage which show all lines, here just return 10 lines. kubectl logs -l app=test --tail=-1 The behavior after fix: Test cases ./kubectl logs -l app=test --tail=-3 ./kubectl logs -l app=test --tail=-1 ./kubectl logs -l app=test --tail=2 ./kubectl logs -l app=test --tail=0
Interesting. I think this also causes to fail: ^-- expected record ^-- some later record (there are only 2 pods running, so is not relevant) ^-- adding makes it work
A bit strange that it finds records a few seconds older as well.. but that's not a big concern, and not relevant for this issue. vs. Although that only happens when the workaround is used: :thinking: P.S.


Question: testKubelet.fakeCadvisor.On("RootFsInfo").Return(cadvisorapiv2.FsInfo{}, nil) kl.nodeLister = testNodeLister{nodes: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: kl.nodeName}}, </del> <ins> { ObjectMeta: api.ObjectMeta{Name: kl.nodeName}, Status: api.NodeStatus{ Allocatable: api.ResourceList{ api.ResourcePods: *resource.NewQuantity(110, resource.DecimalSI), }, }, }, </ins> }} kl.nodeInfo = testNodeInfo{nodes: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: kl.nodeName}}, </del> <ins> { ObjectMeta: api.ObjectMeta{Name: kl.nodeName}, Status: api.NodeStatus{ Allocatable: api.ResourceList{ api.ResourcePods: *resource.NewQuantity(110, resource.DecimalSI), }, }, }, </ins> }} spec := api.PodSpec{NodeName: kl.nodeName, Containers: []api.Container{{Ports: []api.ContainerPort{{HostPort: 80}}}}}

Was there some reason only the checking to the scheduler, not Kubelet?
IIRC it was written at the time when we stated adding checking in Kubelet. In other words - no reason.


Question: }) // CPU tests via replication controllers <del> It(fmt.Sprintf(titleUp, "[Autoscaling Suite]", kindRC), func() { </del> <ins> It(fmt.Sprintf(titleUp, "[Skipped][Autoscaling Suite]", kindRC), func() { </ins> scaleUp("rc", kindRC, rc, f) }) <del> It(fmt.Sprintf(titleDown, "[Autoscaling Suite]", kindRC), func() { </del> <ins> It(fmt.Sprintf(titleDown, "[Skipped][Autoscaling Suite]", kindRC), func() { </ins> scaleDown("rc", kindRC, rc, f) }) })

Per our email discussion while getting 1.1 out the door. It looks like this never happened, though, which means we're not running autoscaling tests against GCE on the 1.1 branch, (we were running it on GKE with the 1.1-features job, but I'm removing that job in ). I'd once again like to (strongly) suggest that we mirror test suites running against release branch to jobs running against master, so that we avoid this problem of accidentally not running tests when we think we are. I'd much rather have a perhaps-unnecessary slow/HPA/whatever suite than continually try to figure out where slow/HPA/whatever tests are running, depending on if I'm looking at release jobs versus jobs running against master. cc
+1 We should be running the autoscaling tests in slow or continuous.


Question: return err } tmpfile := filepath.Join(fstore.path, fmt.Sprintf("%s%s%s", tmpPrefix, key, tmpSuffix)) <del> if err := ioutil.WriteFile(tmpfile, data, 0644); err != nil { </del> <ins> if err := writeFileAndSync(tmpfile, data, 0644); err != nil { </ins> return err } return os.Rename(tmpfile, fstore.getCheckpointPath(key))

The reboot test may cause corrupted checkpoint left on the node. https://k8s- This prevents the sandbox from being stopped properly, so kubelet cannot create a new one. One option is to log and remove the corrupted checkpoint, while cleaning up the pod sandbox resources with best effort (i.e., may leak resources). Another option is to not clean up, and force a human being to intervene and remove the corrupt checkpoint manually. The proper solution perhaps is to scan the checkpoints at startup (or periodically) and repair/remove corrupt checkpoints. /cc related:
/cc
should this block ?
If we can scan the checkpoints at startup time, then repair the corrupted checkpoint, which means, we can recover the state from the runtime, why we introduce the checkpoint at the first place? :-) I am ok with your idea upon the corrupted checkpoints, removing the bad ones as the workaround. Anyway, Kubelet will fire an event of failedSync with the proper message, some manual intervene (the remedy system) can be fired based on the event to resolve the leak resource issue.
I think this shouldn't be the blocker for v1.6, but has an acceptable workaround to address this, and I would like to include that if no one disagreed with that approach.
Adding to milestone to track
Moving this back to 1.6. The checkpoint implementation use , which does not sync at all. No wonder the checkpoints would be corrupted in the reboot tests. I reproduced this issue with a simple go program + reboots on a VM. The problem went away once I forced a I will upload a PR to fix this. /cc


Question: "ServiceAccount tokens. The specified file can contain multiple keys, and the flag can "+ "be specified multiple times with different files. If unspecified, "+ "--tls-private-key-file is used. Must be specified when "+ <del> "--service-account-signing-key is provided") </del> <ins> "--service-account-signing-key-file is provided") </ins> fs.BoolVar(&o.ServiceAccounts.Lookup, "service-account-lookup", o.ServiceAccounts.Lookup, "If true, validate ServiceAccount tokens exist in etcd as part of authentication.")

Within kube-apiserver the flag is referenced in the help string but this flag does not exist kube-apiserver should reference the correct flag Corresponding kubernetes/website documentation issue/PR are: <details</detailsN/A N/A <detailsDownloaded and extracted from release binary </detailsN/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig docs /sig cli


Question: // desiredStateOfWorldPopulatorLoopSleepPeriod is the amount of time the // DesiredStateOfWorldPopulator loop waits between successive executions <del> desiredStateOfWorldPopulatorLoopSleepPeriod time.Duration = 5 * time.Minute </del> <ins> desiredStateOfWorldPopulatorLoopSleepPeriod time.Duration = 1 * time.Minute </ins> ) // AttachDetachController defines the operations supported by this controller.

Problem: An attachable volume referenced by a pod via a PVC object (instead of directly) is not detached if the pod is deleted via namespace deletion (i.e. deleting the pod's namespace instead of the pod object directly). Repro steps: a new volume a new namespace. - PV and PVC objects: - PV/PVC objects are in bound state - pod object that references the volume via PVC - pod gets into running state without issue (i.e. volume is attached) - the namespace - for pod to terminate if volume remains attached after a few minutes Expected: Volume is detached. Actual: Volume remains attached indefinitely. <detailsis: And is: </detailsWorkarounds: the pod object directly before deleting the namespace. In step 7 above do instead If the attachable volume is referenced by a pod using a PVC object, and the pod object is deleted directly (instead of the namespace first), the volume is correctly detached. volume directly in pod (without a PVC object). If the attachable volume is referenced by a pod directly (without a PVC object), namespace deletion results in the volume being correctly detached.
Root Cause: event for pod fails on Attach/Detach controller When a volume referenced via PVC is deleted by namespace, the PV/PVC objects are deleted before the pod object, therefore when the delete event is processed by the attach/detach controller, it fails because it cannot dereference the PVC object. was to ensure volumes are still detached even if a delete event is missed (or fails), however that has a bug in it (the value of is not checked): Proposed Fixes: attach/detach controller to cache PVC/PV so even if it is deleted, it can be recovered on pod deletion events. bug in to check so that it can delete pods even if the delete event is missed (or fails).


Question: }), }, { <ins> desc:      "Single manifest without ID", manifests: api.ContainerManifest{Version: "v1beta1", UUID: "111"}, expected: CreatePodUpdate(kubelet.SET, kubelet.HTTPSource, api.BoundPod{ ObjectMeta: api.ObjectMeta{ UID:       "111", Name:      "111", Namespace: "foobar", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicy{Always: &api.RestartPolicyAlways{}}, DNSPolicy:     api.DNSClusterFirst, }, }), }, { </ins> desc: "Multiple manifests", manifests: []api.ContainerManifest{ {Version: "v1beta1", ID: "foo", UUID: "111", Containers: []api.Container{{Name: "1", Image: "foo"}}},

I'm trying containervm with kubelet built from head and this YAML file: The YAML file is v1beta2 and doesn't set a name to the Pod. When starting kubelet, it doesn't start the "nc" echo server and complains about this on the logs: Chatting with on IM she mentioned it might be due to a change from that now requires non-empty pod names (replaces random generation with validation for non-emptiness) but that means all existing YAML files (even the ones with an older version) will stop working... Is that expected? Thanks! Filipe
Yes, the latest kubelet call validation on all sources: etcd, file, http, etc. now, and PodName is a required field. One potential fix for this is also invoke conversion code before call validation, like what we have done at apiserver. Thoughts?
Just so I get unblocked... Do you have an example YAML file that will set the pod name and will work? Right now this is what I'm using: Should I bump up version to v1beta3 and include a new field somewhere? Thanks! Filipe
The type of that yaml is not pod, and the kubelet code doesn't know how to read pods from a file. So, I think change needs to be reverted before you can make the next ContainerVM release. The v1beta2 vs v1beta3 should not have anything to do with it, unless I'm misunderstanding.
Or will it work to do this:
Still, not sure if you want to release a containerVM version that doesn't accept the old format?
Thanks I made it work with the YAML above! I still have errors in the logs: But the docker containers are being brought up... It's weird that is the YAML field that is missing, considering the source seems to say it's a deprecated field that will be removed at some point... I agree with your point that this needs to be fixed at head before the next containervm release. I opened to increase test coverage of standalone kubelet since we have been seeing many changes break that model recently. Cheers, Filipe
id in v1beta2 maps to name in v1beta3 Tim's change made name required. The internal representation is like v1beta3.

But v1beta3 is not officially stable so we don't want to use it in the GKE or containerVM docs.
The issue is caused by , and affect both v0.8 and v0.9 release for standalone kubelet to support old format ContainerManifests.
Ugh, I thought ID was really required. I spoke with Dawn a bit, and here's my thinking. 1) update all our docs and examples to include ID 2) in file and HTTP config sources, check for Name = "" and apply a default, maybe log a warning that this is deprecated 3) wait for the ContainerManifest -Pod conversion to really make it required


Question: <ins> FROM scratch ADD hello / EXPOSE 8080 ENTRYPOINT ["/hello"] </ins>

/kind feature Hey, in K8s 1.23 we are planning to deprecate multiple klog flags. Deprecated flags: , , , , , , , , , , , One of the alpha milestones is requires us to remove usage of deprecated flags within K8s repo. Examples found for kubemark [x] [x] [x] [x] Recommendation, either: Stop depending on writing to files and remove flags , or Add go-runner to kubemark image and migrate to /sig scalability /wg structured-logging
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Resolved in both k/k and kubernetes-sigs/cloud-provider-azure. /close


Question: instanceCache instanceCache <ins> clientBuilder    controller.ControllerClientBuilder kubeClient       clientset.Interface eventBroadcaster record.EventBroadcaster eventRecorder    record.EventRecorder </ins> // We keep an active list of devices we have assigned but not yet // attached, to avoid a race condition where we assign a device mapping // and then get a second request before we attach the volume

On AWS when we call "attach" of a volume then we periodically we check (via API call) if volume has indeed been attached. We run the check in a loop with timeout of 21 minutes. AWS documents that - if a volume is stuck in "attaching" state for too long, the node where volume is being attached is in "dirty" state and must be rebooted to fix the problem. Until node is rebooted, more pods with volumes to same node is only going be stuck in same state (i.e "attaching"). We propose that - if attach operation times out waiting for volume in "attaching" state, we should simply taint the node so as it can not affect other pods and a admin can manually fix the node. /sig aws /sig storage cc


Question: continue } // in this scenario the vmSetName is an empty string and the name of vmss should be obtained from the provider IDs of nodes <del> vmssName, resourceGroupName, err := getVmssAndResourceGroupNameByVMProviderID(node.Spec.ProviderID) </del> <ins> resourceGroupName, vmssName, err := getVmssAndResourceGroupNameByVMProviderID(node.Spec.ProviderID) </ins> if err != nil { klog.V(4).Infof("ensureVMSSInPool: found VMAS node %s, will skip checking and continue", node.Name) continue

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When using Internal Standard LoadBalancer on Azure, although VMSS instances have been to LB backend address pools, the VMSS itself is not . Hence, if customer upgrades these VMSS instances to latest model, the backend address pool would be lost and hence the LoadBalancer services would be broken. What you expected to happen: Azure VMSS is also to internal SLB backend address pool automatically. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /kind bug /priority critical-urgent /area provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --What happened: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /priority critical-urgent /area provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure
Can you share some insights when this could be get fixed?


Question: "time" autoscaling "k8s.io/api/autoscaling/v2beta2" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/sets"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the HPA scales up beyond the capacity of the cluster, many unscheduled pods are created. When load is removed, the usage ratio drops and the HPA tries to scale down. The unscheduled pods do not produce metrics, so the HPA fills in the 100% usage and re-evaluates the usage ratio. Because of the many fake 100% metrics, the new usage ratio is above the target value and the direction of scale is reversed. In this case, the HPA decides not to make any change to the current scale. What you expected to happen: I would expect the HPA to not get stuck when load is removed. How to reproduce it (as minimally and precisely as possible): and put it under load in a cluster without cluster autoscaling. the Deployment to increase the CPU request beyond the capability of the cluster (e.g. 500m). This should produce many Pending pods. load. The Pending pods will keep the HPA scaled indefinitely. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.9-gke.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T21:30:53Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig autoscaling
This might Previous I suspected that this was the same issues as (HPA positive feedback loop).

However it turns out to be different, that Pending pods cause HPA to get stuck.
Original reason for using 100% utilization for missing metrics when scaling down:


Question: verifyResult(cs, 0, 1, ns) }) <del> It("validates that a pod with an invalid NodeAffinity is rejected", func() { By("Trying to launch a pod with an invalid Affinity data.") podName := "without-label" _, err := cs.CoreV1().Pods(ns).Create(initPausePod(f, pausePodConfig{ Name: podName, Affinity: &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{}, }, }, }, }, }, })) if err == nil || !errors.IsInvalid(err) { framework.Failf("Expect error of invalid, got : %v", err) } }) </del> /* Testname: scheduler-node-selector-matching Description: Ensure that scheduler respects the NodeSelector field

SchedulerPredicates [Serial] validates that a pod with an invalid NodeAffinity is rejected Has been failing consistently since 4/17. Here are the /kind bug /priority failing-test /priority important-soon /sig scheduling /milestone 1.11 cc /assign for triage.
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : ## Failing Job SchedulerPredicates [Serial] validates that a pod with an invalid NodeAffinity is rejected /priority failing-test /priority important-soon /sig scheduling /milestone 1.11 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.11
/status approved for milestone
Also fails as part of upgrade job as seen below
Also fails as part of following jobs https://k8s- https://k8s- https://k8s- https://k8s-
xref
/assign
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 6 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Seems like this is failing in (just with slightly different versions)


Question: cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" "k8s.io/kubernetes/pkg/kubectl/resource" "k8s.io/kubernetes/pkg/runtime" <ins> "k8s.io/kubernetes/pkg/util/validation" </ins> ) // SelectorOptions is the start of the data required to perform the operation.  As new fields are added, add them here instead of

Is this a BUG REPORT or FEATURE REQUEST? (choose one):BUG REPORT Kubernetes version (use ):Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2610+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-12-29T06:28:44Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2610+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-12-29T06:28:44Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"} What happened:
Could someone from please help with triage. Thanks!
There is already a PR on the way.


Question: config.DeleteNodePortService() ginkgo.By(fmt.Sprintf("dialing(http) %v (node) --> %v:%v (nodeIP) and getting ZERO host endpoints", config.NodeIP, config.NodeIP, config.NodeHTTPPort)) <del> err = config.DialFromNode("http", config.NodeIP, config.NodeHTTPPort, config.MaxTries, config.MaxTries, sets.NewString()) </del> <ins> // #106770 MaxTries can be very large on large clusters, with the risk that a new NodePort is created by another test and start to answer traffic. // Since we only want to assert that traffic is not being forwarded anymore and the retry timeout is 2 seconds, consider the test is correct // if the service doesn't answer after 10 tries. err = config.DialFromNode("http", config.NodeIP, config.NodeHTTPPort, 10, 10, sets.NewString()) </ins> if err != nil { <del> framework.Failf("Error dialing http from node: %v", err) </del> <ins> framework.Failf("Failure validating that node port service STOPPED removed properly: %v", err) </ins> } })

gce-master-scale-correctness Kubernetes e2e suite.[sig-network] Networking Granular Checks: Services should update nodePort: udp [Slow] ci-kubernetes-e2e-gce-scale-correctness.Overall kubetest.Test Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition Kubernetes e2e suite.[sig-network] Services should be able to update service type to NodePort listening on same port number but different protocols 12/1/2021 17:36:39 ET No response No response /sig storage /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.23 /priority important-soon
/assign aojea
/assign bowei
heh, what are the chances, two different test services took the same port The test was expecting to not receive any answer but the second service replies and makes the test fail link


Question: # disallow transitive dependencies on k8s.io/kubernetes loopback_deps=() kube::util::read-array loopback_deps < <(go list all 2>/dev/null | grep k8s.io/kubernetes/ || true) <del> if [[ -n "${loopback_deps[*]}" ]]; then </del> <ins> if [[ -n ${loopback_deps[*]:+"${loopback_deps[*]}"} ]]; then </ins> kube::log::error "Disallowed ${repo} -> k8s.io/kubernetes dependencies exist via the following imports: $(go mod why "${loopback_deps[@]}")" exit 1

error in update vendor after running Here is how you can repro on current head: 1- ./hack/pin- 2- hack/update- the test for empty bash array is broken, since bash considers it as an unset var.
/assign /sig release
/lifecycle active


Question: import ( "context" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : In block volume tests, wait for the Pod to be deleted instead of letting it be handled by the framework's namespace deletion Why is this needed: I think the regression caused by could have been caught if the tests check for Pod deletion because the Pod object shouldn't be deleted if TearDown fails.
/assign
This issue is not specific to block tests. We should in ideally wait for pod deletion at the end of all types of tests to confirm cleanup logic is functioning
I checked the tests under test/e2e/storage/testsuites/ where DeletePodWithWait is used to wait for the deletion.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten is this still an issue? One way to check is to make a breaking change in cleanup and see if the pull job fails.
I went through all block volume usage in e2e tests: All of them but one wait for their pods to be deleted. I'm fixing the remaining test in


Question: import ( "context" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: [sig-storage] PersistentVolumes GCEPD should test that deleting the Namespace of a PVC and Pod causes the successful detach of Persistent Disk Testgrid link: Reason for failure: Anything else we need to know: it seems like this started flaking more frequently during 2019-12-02 triage link: examples of failures: /area platform/gcp /sig storage
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : <!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --- pull-kubernetes-e2e-gce [sig-storage] PersistentVolumes GCEPD should test that deleting the Namespace of a PVC and Pod causes the successful detach of Persistent Anything else we need to know: it seems like this started flaking more frequently during 2019-12-02 triage link: examples of failures: /sig storage Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/gcp
/sig cloud-provider I question whether we should be running this test by default, since it seems like it's exercising a specific cloud provider's implementation of persistent volumes?
Another occurrence of the test failure:
interestingly this is not flaky at all against master (opening the last several dozen runs doesn't show any failures/retries of that test) and is exceptionally flaky in PR jobs: in the last 47 runs, there were 26 failures (and only one PR looked like it could have introduced relevant failures): (potentially PD storage-related PR) (x3) (x3)
/assign
also, the timeout error is on the namespace delete, not the PD detach
opened to mark flaky to unblock the queue, opened to add debugging around the timeout failure; will keep this issue open and in the milestone to ensure the test is fixed or removed before 1.18.0
Looking at this run: namespace deletion timeout in the test is 5 minutes, so looks like this took just a little longer.

Looking into what is taking the pod so long to be deleted
It seems like kubelet tore down and deleted the Pod pretty quickly and it was registered by apiserver log. For some reason, the namespace controller doesn't see that deletion?
From in , a little earlier than the log posted: E1216 11:23:16. 1335 ] Operation for """ ("-c02f-4701-9708-")" failed. No retries permitted until 2019-12-16 11:23:16. +0000 UTC m=+1060. (durationBeforeRetry 500ms). Error: "recovered from panic "runtime error: invalid memory address or nil pointer dereference". (err=<nil) Call stack:ngoroutine [running]:(0xc0032cad50)n +0xb5npanic(0x3a27160, 0x6d2bdf0)n +0x1b2(0xc000c94a20, 0x81, 0x0, 0x0, 0x4979460, 0xc0021d0b40, 0x0, 0x0)n +0xbd(0xc001f82a80, 0x7f, 0x4a33500, 0xc002099790, 0xc0011a7ba8, 0x0, 0x7f)n +0x20cnpath/filepath.Walk(0xc001f82a80, 0x7f, 0xc001fdcba8, 0xc001f82a80, 0x7f)n +0xff(0x4a28680, 0xc00090d580, 0xc0016416c0, 0x3a, 0xc002b50dd0, 0xb, 0xc0016416c0, 0x3a)n +0x8ce(subpath).CleanSubPaths(0xc0004a08b0, 0xc0016416c0, 0x3a, 0xc002b50dd0, 0xb, 0x1, 0x0)n +0x5f(operationGenerator).

GenerateUnmountVolumeFunc.func1(0xc0012f95e0, 0xc0032cad50, 0x2e, 0x1)n +0x15a(GeneratedOperations).Run(0xc001dd3200, 0x45c1e0, 0xc0016b6ba0, 0xc000e16c90, 0xc0019c02a1)n +0xd6(nestedPendingOperations).Run.func1(0xc001039580, 0xc0010a2e10, 0x48, 0xc0020085d0, 0x24, 0xc001dd3200, 0x0, 0x0, 0x0, 0x0)n +0x114ncreated by (*nestedPendingOperations).Runn +0x39fn" which is addressed over
I'm not sure the kubelet panic is related to this test flake. So far in this flake, the API server did register pod deletion, but the namespace controller still seems to think the pod is not deleted.
This problem also happened in kops.
I encountered this problem in kubernetes and Kops projects, which seems to be more serious.
test has been marked flaky, and can now be seen in root cause is an extremely slow namespace deletion controller; see analysis in
Opened to track the namespace controller issue. In the meantime, I would probably suggest rewriting this test to: the namespace delete deletecollection calls against the pvcs and pods, rather than waiting for the namespace controller for the absence of the pvc and pod objects, rather than the namespace
I think then the test case would be very similar to "[sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node", which 2 PVCs used by a single Pod the pod reschedules the pod to a different node and validates that it was successful (which indicates that it got detached and reattached successfully) Pod and PVC at the end and verifies PV deletion.

I'm in favor of just removing this test (and having a separate non-storage test for namespace deletion) since we already have similar coverage in an existing test case.
Doing this inside a deleting namespace ensures nothing in the PVC in mount or pod teardown blocks on the inability to create events. I would keep that aspect of it and just trigger the pod/pvc deletions yourself.
Ok, then I would be more in favor of adding a new test case in our testsuite that explicitly does the namespace deletion. I want to start moving away from having cloud provider-specific tests, and as a bonus we'll get coverage for CSI drivers too.
I'm ok adding a non-storage test specifically around namespace deletion, but ensuring the PV/PVC infrastructure behaves well in a namespace that is being deleted seems valuable as well (and that test would not need to wait on the namespace to disappear)
Agree. When I said "testsuite" I meant the
/milestone v1.19
Hit by see


Question: currState, ok := cache.podStates[key] if ok && currState.pod.Spec.NodeName != pod.Spec.NodeName { <del> return fmt.Errorf("pod %v state was assumed on a different node", key) </del> <ins> return fmt.Errorf("pod %v was assumed on %v but assigned to %v", key, pod.Spec.NodeName, currState.pod.Spec.NodeName) </ins> } switch {

/kind bug There are several error messages in the scheduler cache that are confusing to read, at least one of these surfaced up to end users via Events. I noticed these two in particular but there may be others: Whether or not these should actually be surfacing as events, the messages should be cleaned up.
cc
Seems that these are internal or implement details in scheduler, do our end users can fully understand what the events really mean? I think it is depend on our own implement. idea?
I believe, the problem is with in the message but I am not able to reproduce it, can you please share steps to reproduce this.
No unfortunately, one of our users reported seeing it when there system was in a bad state, not in a reproducible way. The point of the issue is just that the grammar in those messages is bad, which should be fixable even if you can't get them to fire.


Question: // Invalidate the cache right after updating defer as.cloud.vmCache.Delete(vmName) <del> _, err = as.VirtualMachinesClient.CreateOrUpdate(ctx, nodeResourceGroup, vmName, newVM, "attach_disk") </del> <ins> _, err = as.VirtualMachinesClient.Update(ctx, nodeResourceGroup, vmName, newVM, "attach_disk") </ins> if err != nil { klog.Errorf("azureDisk - attach disk(%s, %s) failed, err: %v", diskName, diskURI, err) detail := err.Error()

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Instead of using the VM CreateOrUpdate () function of the Azure Go SDK, use the VM Update function () in the azureclient () Why is this needed: Currently, the Azure Cloud Provider is using the VM CreateOrUpdate call when e.g. detach/attaching disks to a VM. In some scenarios, this will re-create a VM that was deleted by an external application. This can lead to orphaned VMs or blocked resources. The Update function will only update an existing VM and will fail if the VM does not exist.
/sig azure
/assign
/sig cloud-provider
Could you describe more about the scenarios? Have you noticed such issues? I wonder adding etags to the API call could help to prevent this?
If you have an application that maintains your K8s clusters, e.g. takes care of OS update/patching. If the application wants to patch the OS of a worker node, it would drain the node first. If draining times out, it will go ahead and delete the VM anyway. But K8s might still be detaching disks. These detach operations, because they use CreateOrUpdate, will recreate the VM. Is there a reason why you would want to create a VM i.e. why CreateOrUpdate is required and Update would not work? AFAIK, Azure Compute RP does not support etags.
that sounds reasonable, thanks change to use function would be safer since it's only detach/attach disk operation. I will do that.
thanks for the information. Reasonable enough. /assign
/milestone v1.16
BTW, the reason why we only use func in the beginning is that Azure Compute go sdk did not provide a func at that time...
Great. Thanks!


Question: // KubeletExtraArgs passes through extra arguments to the kubelet. The arguments here are passed to the kubelet command line via the environment file // kubeadm writes at runtime for the kubelet to source. This overrides the generic base-level configuration in the kubelet-config-1.X ConfigMap // Flags have higher priority when parsing. These values are local and specific to the node kubeadm is executing on. <ins> // A key in this map is the flag name as it appears on the // command line except without leading dash(es). </ins> KubeletExtraArgs map[string]string // IgnorePreflightErrors provides a slice of pre-flight errors to be ignored when the current node is registered.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading the doc about NodeRegistration, with an interest in specifying some . I noticed that the doc (e.g., ) does not tell me exactly how to identify each arg: exactly as on the command line (including leading dash(es)), as on the command line but without leading dash(es), or camelCase. I expected the doc to enable me to write some with confidence. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
FWIW the answer is flag names as on the command name but without any leading dashes. This also applies to controllerManager.extraArgs, scheduler.extraArgs, etc. which hopefully should all cover this similarly (if not now, after fixing ...)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading about how to use kubeadm and noticed a typo in the API documentation. speaks of the "node subnet" where I think "pod subnet" is intended. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/area kubeadm


Question: "io" "net" "net/url" <del> "os" </del> "regexp" "strconv" "strings"

I am having trouble on my AWS 1.3.0beta0 cluster to attach EBS with the controller-manager based attacher. There problem is here: This is failing as the kube-controller-manager is running in a pod via kubelet manifests. I think if you have a cluster with different instance types mixed you ran into trouble as well, as the device names will be either (sdX or xvdX) and the cluster will do it based on the state of the master (controller-manager). My work around for now is to enable a host mount of for the controller manager. But this need to be addressed at some point. Is there any way we get the correct name from the AWS api? I think it's depending on the kernel version if it is xvdX or sdX. Am I right on this? A more ugly solution could be to annotate the node during registration, if it's has or device names maybe you can help me get a proper solution for this. I am happy to contribute a PR
I have just found , I think we could get this information from AWS:
That code is supposed to check, but it is quite possible I got it backwards looking at it. Also I agree that the logic is going to be problematic anyway with mixed instance types etc if we've moved attachment to KCM. What AMI are you using? I may have just been "lucky" so far in the AMIs I've used, in that the heuristics have worked correctly. Also, if you have time, would be great to know: /dev/ looks like on one of the problematic nodes (i.e. in particular we have /dev/sda and/or /dev/xvda)? is the output of the describe-images command for that AMI (particularly if it is private)? I like the describe-images approach!

I'm marking this as 1.3 P1 (sorry as for now my working hypothesis is that this was introduced when we moved disk attachment to KCM (I believe previously the kubelet did it, but want to double-check)
I tried to get a such environments with CoreOS amis: (all eu-west1) vtype is PV vtype is HVM The describe instances returns different rootdevicenames: But if I ssh into the two instances, the devices are called the same for both: I think we have to somehow signal the devicename from the Kubelet to the KCM to solve this properly. My understanding is to get the sdX names we have to use an pretty old kernel. Maybe it'a good idea to change the default behaviour, see PR . So fallback to '/dev/xvdX'
I don't see any indication that this is a regression. I am kicking it out of milestone 1.3. Explain how it is a regression or otherwise super-important to get it back on the milestone.
It is a regression: volume mounting is broken on AWS. My believe is that it happens because we have moved volume mounting to KCM, and/or KCM runs in a container. Between those two, volume mounting just doesn't work right now.


Question: "fmt" "io" "net" <ins> "net/http" </ins> "os" "os/exec"

Environment: VMware CentOS Linux release 7.2.1511 (Core) kernel: 3.10.0-327.36.1.el7.x8664 Installed packages: kubeadm.x8664 1.5.0-0.alpha.0.1534.gcf7301f kubectl.x8664 1.4.0-0 kubelet.x8664 1.4.0-0 kubernetes-cni.x8664 0.3.0.1-0. docker-engine.x8664 1.11.2- docker-engine- 1.12.1- What happened: kubeadm init hangs indefintely after: kubeadm init <master/tokensgenerated token: "." <master/pkicreated keys and certificates in "/etc/kubernetes/pki" <util/kubeconfigcreated "" <util/kubeconfigcreated "" <master/apiclientcreated API client configuration <master/apiclientcreated API client, waiting for the control plane to become readykubeadm init What you expected to happen: Master node initialize Anything else do we need to know: Only the kubeadm and kublet processes are running. When I do docker ps no containers are running. If i type docker images the only image that has been downloaded are: I've also tried: kubeadm init --api-advertise-addresses but with the same outcome (followed the clean-up at ) Im behind a firewall so I'm using proxy settings for docker. I have confirmed I can reach images on docker hub and google cloud registry
will download the required binaries viz etcd. When you are behind firewall kubeadm will hang/wait forever to download the binaries. I dont know whether it is addressed correctly in kubeadm implementation. I am also facing the same issue and waiting for the resolution.
Thx, I suspected it to be some firewall issue. Do you know if there's a way to add httpsproxy & httpproxy settings to kubeadm?
Kubeadm doesn't seem to pick proxy environment variables. I am looking at recompiling with proxy support may work. But I dont know how to do that. Waiting for updates about proxy
This issue same as Lets join there. The solution mentioned there may work.

You can try kubeadm with patch applied from PR : It works at least in my environment behind restrictive proxy.
thanks, I will look into it


Question: func (detacher *azureDiskDetacher) UnmountDevice(deviceMountPath string) error { err := util.UnmountPath(deviceMountPath, detacher.plugin.host.GetMounter(detacher.plugin.GetPluginName())) if err == nil { <del> glog.V(4).Infof("azureDisk - Device %s was unmounted", deviceMountPath) </del> <ins> glog.V(2).Infof("azureDisk - Device %s was unmounted", deviceMountPath) </ins> } else { <del> glog.Infof("azureDisk - Device %s failed to unmount with error: %s", deviceMountPath, err.Error()) </del> <ins> glog.Warningf("azureDisk - Device %s failed to unmount with error: %s", deviceMountPath, err.Error()) </ins> } return err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Current logging is not enough for azure disk diagnostics, change some log level from 4 to 2 What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign


Question: var _ = framework.KubeDescribe("Summary API", func() { f := framework.NewDefaultFramework("summary-test") Context("when querying /stats/summary", func() { <ins> AfterEach(func() { if CurrentGinkgoTestDescription().Failed && framework.TestContext.DumpLogsOnFailure { framework.LogFailedContainers(f.ClientSet, f.Namespace.Name, framework.Logf) } }) </ins> It("should report resource usage through the stats api", func() { const pod0 = "stats-busybox-0" const pod1 = "stats-busybox-1"

https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
The test failed because the busybox container was crash-looping, but I'm not sure why.
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
- Any ideas what changed in the past week that would cause the Kubelet usage/working set memory usage to suddenly increase? It may be a non-issue, but it's concerning that this seems to have recently changed. - This is only happening on GCI. Did we change the GCI image being tested recently?
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
Last image version update for testing was 11 days ago () though we are soon going to update it again ().
Since we increased the bound on (), we should also increase the bound on ( is usage - total inactive file). I think this may be the cause of this flake.
I agree that the bounds should match, but the cause of both flakes (why usage & workingset suddenly increased) is still unknown.
Usage workingset so it may not be necessary to bump both ;) I think this may be the caused by the gci mounter.

/cc
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
I checked the log from the last failure, didn't see anything related to gci mounter. The test failed on trusty node which mounter should not be used at all.
Yes, the latest failures are just a flake introduced by The bounds need to be adjusted for the misc container. I'll send a PR soon.


Question: } } <ins> func TestPrintNodeExternalIP(t *testing.T) { printer := NewHumanReadablePrinter(PrintOptions{ ColumnLabels: []string{}, Wide:         true, }) table := []struct { node       api.Node externalIP string }{ { node: api.Node{ ObjectMeta: api.ObjectMeta{Name: "foo1"}, Status:     api.NodeStatus{Addresses: []api.NodeAddress{{Type: api.NodeExternalIP, Address: "1.1.1.1"}}}, }, externalIP: "1.1.1.1", }, { node: api.Node{ ObjectMeta: api.ObjectMeta{Name: "foo2"}, Status:     api.NodeStatus{Addresses: []api.NodeAddress{{Type: api.NodeInternalIP, Address: "1.1.1.1"}}}, }, externalIP: "<none>", }, { node: api.Node{ ObjectMeta: api.ObjectMeta{Name: "foo3"}, Status: api.NodeStatus{Addresses: []api.NodeAddress{ {Type: api.NodeLegacyHostIP, Address: "1.1.1.1"}, {Type: api.NodeExternalIP, Address: "2.2.2.2"}, {Type: api.NodeInternalIP, Address: "3.3.3.3"}, {Type: api.NodeExternalIP, Address: "4.4.4.4"}, }}, }, externalIP: "2.2.2.2", }, } for _, test := range table { buffer := &bytes.Buffer{} err := printer.PrintObj(&test.node, buffer) if err != nil { t.Fatalf("An error occurred printing Node: %#v", err) } if !contains(strings.Fields(buffer.String()), test.externalIP) { t.Fatalf("Expect printing node %s with external ip %#v, got: %#v", test.node.Name, test.externalIP, buffer.String()) } } } </ins> func contains(fields []string, field string) bool { for _, v := range fields { if v == field {

Nice to have for nodePort service debugging, creating manual firewalls, hitting hostNetwork or hostPort pods etc. Currently I need to either -o yaml, --template, or switch to gcloud.
This should be quite easy, should be good for contribution.
I could take a look at this. What information exactly that is available in yaml output should be shown? This is example yaml from my node: There should be an additional address with type , right? Should all types be supported or only external when is used?
-o wide should just dump what we get from the api-server right? What changes should be in kubectl?
-o yaml dumps apiserver outout, -o wide appends useful information at the cost of overflowing 80 chars. I'm asking that -o wide does for nodes what it already does for pods. I currently have such a function in my bash rc: I use it pretty often. That's a signal that more people are probably doing the same.
thanks!


Question: close(heartbeat.closer) t.Errorf("unexpected heartbeat timeout") } <ins> wg.Wait() </ins> } func TestWebSocketClient_StreamsAndExpectedErrors(t *testing.T) {

FAIL: TestWebSocketClientHeartbeatSucceeds also flake in ci-kubernetes-unit use of closed network connection /cc /sig api-machinery network cli
/cc /triage accepted
seems it got better since
The new flake in this test is like I also see the failure in some PR: for instance.
yeah, it is racing somewhere , at least we have a different symptom now
/help someone want a great challenge and make a very useful contribution?
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
something odd is with the test, the panic is because it logs on the goroutine, but I'm not completely sure the test is working as expected
my apologies, I found the problem and I'm submitting a patch /assign /unassign you can help me with the review if you want


Question: } defer os.Remove(f.Name()) <ins> if _, err := serviceaccount.ReadPublicKeys(f.Name()); err == nil { t.Fatalf("Expected error reading keys from empty file, got none") } </ins> if err := ioutil.WriteFile(f.Name(), []byte(rsaPublicKey), os.FileMode(0600)); err != nil { t.Fatalf("error writing public key to tmpfile: %v", err) }

Kubernetes version Serverversion 1.5.2 (running Hyperkube v1.5.2coreos.1 ()) Environment: Bare-Metal Coreos 1235.6.0 (Kernel 4.7.3-coreos-r2) What happened: kube-controller-manager with ecdsa key fails to start F0127 09:59:04. 1 ] error running controllers: Error reading key for service account token controller: error reading private key file data does not contain a valid RSA or ECDSA private key The key is valid (and works with e.g. the api-server) Command used to create the key:
the issue is the block. removing that (invoke openssl with to omit it from the start) will allow the key to be consumed. will fix in 1.6 to ignore that block when reading the private key block