Question: // secrets. func (kl *Kubelet) getPullSecretsForPod(pod *v1.Pod) []v1.Secret { pullSecrets := []v1.Secret{} <ins> failedPullSecrets := []string{} </ins> for _, secretRef := range pod.Spec.ImagePullSecrets { if len(secretRef.Name) == 0 {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --imagePullSecrets can to a pod manifest to reference a secret containing registry credentials. Currently, if you add the name of a secret that does not exist, there is no warning or error. I would expect there to be an error and a failed deployment if the secret does not exist/cannot be found in the namespace. If people are pulling public images, they may still wish to authenticate to a registry, for example dockerhub, before pulling the image, in order to utilise their subscription account rate limits. Currently if they use imagePullSecrets and the referenced secret does not exist, they may assume they are authenticating to their account when they are not, and will still be able to pull public images without any warning that authentication is not taking place. Add to a pod spec, and pull images from
/sig node (based on the field location)
(and/or) /sig auth
correct me if not. Pulling private images with nonexists secret, pod will turn to . However, pulling public image with nonexists secret, no error will display yet.
Similar issue with .
Yes that's correct.
I think so. Working on this later. Will also live up to your expectations, right?
But I check it already have a warnging events if secret not exist and pullImageFail: line:147
Now, only info message when pullsecret not exist
maybe direct return and not enter is better. if pullSecret is not existed
if you pull image from a public repo, the process should not be impacted even no secrets exists. And it's what it is right now.
I think a warning event like could be here /good-first-issue /triage accepted
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/assign
GitHub didn't allow me to assign the following users: EricWvi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: EricWvi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : I would like work on this. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to work in this. /assign
Thanks for wanting to jump in on this, but it's not clear this is a good first issue… making the proposed change would break deployments/pods that are currently working/running, so it's unlikely this would be acceptable from a compatibility perspective.
Requiring image pull secrets to exist would have to be opt-in, which means an API change. I don't think this is worth the addition, but would like to hear sig-node's perspective. An alternative could be to explore ways to make it more obvious to the user when this situation was occurring without automatically failing the pod
hi , we will not fail the pod, we just want to add some warning to make it obviously, maybe it's the same as you advised. correct me if not.
To be clear here, only adding a warning is marked as good first issue here, not changing any pod behaviour.
gotcha, that seems more reasonable, thanks for clarifying as a note, make sure the warning events are generated at most once in that function, rather than once per missing secret object (if there are multiple missing secrets)
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed .

If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, a warning would be good. I feel an error would have be better, but as has already been mentioned above we are too far along now for this change as it could break some users existing deployments, so a warning makes the most sense now.
/assign
based on what you have stated above, I see an INFO log already exists! If I'm understanding this correctly, now we need to raise a warning event in the same code block to fulfill this request. Is that right?
I think has created a above and labeled /lgtm already.
Hey I don't know anything about how to code but i am interested to contribute on kubernetes So help me to contribute
/cc
there is PR already here , just needs approval
it was just FYI =) Reviewed the PR, thank you
Thanks for the review :)
/assign
will like to work on it. /assign
Hey , would love to work on this issue
there is already a PR: do you plan to push the PR forward?
I have been busy with other things recently, if anyone is willing to continue this problem. I will close the PR.
if noone else is working on this task, I would like to take this up.
Feel free to work on this , you can refer to for some advices.
Thanks! Will refer it
/assign
/assign
/assign
Is this issue still open/active?
Hello I was working on it but not able to continue it due to some personal reasons. You can take over if you would like to. You can refer this PR if you want to.
Yes sure!!!
Can I try this if no one is available to work on it ?
Yes go on!!!
/assign
Sorry for keep opening the PR. something went wrong on my fork branch. I eventually needed to re-fork the repo so it caused the PR closed automatically
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Can I give it a go if no one is working on it?
Thanks for the offer, but it looks like was open and had an update that was waiting for review (which I didn't realize)
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
is this issue still available to pick up?
/remove-lifecycle stale
Hi, is this issue still available?
Hi! It feels that this issue is stalled (although it lacks the label) and the related PRs have been abandoned. I'll work on it if that's ok /assign
As well as an event, we might like to directly report a warning to end users (suggested in ). More complexity though.
should it be part of this issue? it only mentions but as you guys (the contributors) decide. is based on which was labelled as and suggested as a starting point before being closed.


Question: }) }) }) <ins> ginkgo.Context("when querying /metrics", func() { ginkgo.BeforeEach(func() { // ensure APIs have been called at least once endpoint, err := util.LocalEndpoint(defaultPodResourcesPath, podresources.Socket) framework.ExpectNoError(err) cli, conn, err := podresources.GetV1Client(endpoint, defaultPodResourcesTimeout, defaultPodResourcesMaxSize) framework.ExpectNoError(err) defer conn.Close() _, err = cli.List(context.TODO(), &kubeletpodresourcesv1.ListPodResourcesRequest{}) framework.ExpectNoError(err) _, err = cli.GetAllocatableResources(context.TODO(), &kubeletpodresourcesv1.AllocatableResourcesRequest{}) framework.ExpectNoError(err) }) ginkgo.It("should report the values for the podresources metrics", func() { // we updated the kubelet config in BeforeEach, so we can assume we start fresh. // being [Serial], we can also assume noone else but us is running pods. ginkgo.By("Checking the value of the podresources metrics") matchResourceMetrics := gstruct.MatchKeys(gstruct.IgnoreExtras, gstruct.Keys{ "kubelet_pod_resources_endpoint_requests_total": gstruct.MatchAllElements(nodeID, gstruct.Elements{ "": timelessSampleAtLeast(1), }), "kubelet_pod_resources_endpoint_requests_list": gstruct.MatchAllElements(nodeID, gstruct.Elements{ "": timelessSampleAtLeast(1), }), "kubelet_pod_resources_endpoint_requests_get_allocatable": gstruct.MatchAllElements(nodeID, gstruct.Elements{ "": timelessSampleAtLeast(1), }), // not checking errors: the calls don't have non-catastrophic (e.g. out of memory) error conditions yet. }) ginkgo.By("Giving the Kubelet time to start up and produce metrics") gomega.Eventually(getPodResourcesMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) ginkgo.By("Ensuring the metrics match the expectations a few more times") gomega.Consistently(getPodResourcesMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) }) }) </ins> }) func requireLackOfSRIOVDevices() {

In commit we podresources-related metrics, to make sure the feature was monitoring-friendly. Because of a implementation bug, the metrics have been , they are updated correctly, but they are never registered, so they don't show up when doing like the podresources related metrics should be reported in the endpoint in the kubelet the kubelet, even the simplistic environment of the tests is sufficient sure to enable the and feature gates (on by default in kube = 1.25 at least) the podresources endpoint issuing and calls. The actual output is not relevant the metrics: The bug was introduced in the original commit <details</details<detailsNot relevant. </details<detailsNot relevant </details<detailsNot relevant </details<detailsNot relevant </details<detailsNot relevant </details
/sig node /assign
/cc /triage-accepted
/triage accepted /cc


Question: // catch race conditions introduced by callers updating pod status out of order. // TODO: have KillPod return the terminal status of stopped containers and write that into the //  cache immediately <del> podStatus, err := kl.containerRuntime.GetPodStatus(ctx, pod.UID, pod.Name, pod.Namespace) </del> <ins> stoppedPodStatus, err := kl.containerRuntime.GetPodStatus(ctx, pod.UID, pod.Name, pod.Namespace) </ins> if err != nil { klog.ErrorS(err, "Unable to read pod status prior to final pod termination", "pod", klog.KObj(pod), "podUID", pod.UID) return err } <ins> preserveDataFromBeforeStopping(stoppedPodStatus, podStatus) </ins> var runningContainers []string type container struct { Name       string

My pod completes (success or failure). The IP temporarily is removed on one of the status updates. The IP is persistent . Apply: Watch: The initial 'succeeded' should have the IP still. On older versions: Full pod state, from 1.27: <details</detailsPrior discussion many, see above Tested on kind Linux <details</details<details</details<details</details
/sig network maybe Relevant PRs in 1.27:
cc
/priority important-soon
is about endpointslices, so no related seems related , ping
I can confirm this commit specifically is the culprit (when I build and test the parent commit it works ok): For now I don't know the reason, cc in case you have some ideas. EDIT: when commenting out these it works as before. I suspect this is because the does not have the information which is determined in
I've opened a PR with a fix. PTAL: Additionally, it turns out that the bug causes one more unnecessary request with status update.
This also relates to sig-node: /sig node We talked about this in the SIG Network meeting, looks like you are taking a look: /assign Please let us know what you find, and please let us know if you need any support!
Thank you! I think I have got to the bottom and summarized the findings in the PR description: I would love to move forward with the review :)
the question from SIG Node meeting was - what is the impact of it besides the flickering IP on status. This may help justify the cherry-pick.
do you have some additional insights about the impact?
In our usage we had to handle events on a pod informer differently to account for the fact we may read it in an invalid state if we read at the wrong time. So for us technically it hsa no impact since we worked around it, for others it could be quite severe potentionally
Thanks! Just FYI as demonstrated by experiments on you could also face this situation in case of kubelet restart. So, probably good you put this workaround in place anyway.
/triage accepted People affected by this regression likely will be affected by kubelet restart in a very precise moment. But it is a low probability event so I would advocate to fix the regression by backporting this fix instead of forcing consumers to work around this behavior.


Question: import ( "encoding/json" "fmt" <del> "reflect" </del> "time" corev1 "k8s.io/api/core/v1" <ins> apiequality "k8s.io/apimachinery/pkg/api/equality" </ins> apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" utilruntime "k8s.io/apimachinery/pkg/util/runtime"

Please write a test that: Starts kube-apiserver Adds a webhook which blocks everything it is possible to block. Re-starts kube-apiserver Verifies that kube-apiserver comes back up Verifies that it can remove the broken webhook. This would have caught
cc
Hi I'm new to kubernetes but eager to contribute. I think this will be a good first issue for me but I'll need some guide on how to proceed I will need the first two points explained and also which file I'm supposed to be making my changes to.
Here is a good example of how to start a realistic kube-apiserver in integration tests:
"Verifies that it can remove the broken webhook." sounds dangerous. The apiserver should come up and it should be possible to remove a webhook via the API. But removing a potentially broken (or intentionally DoS'ed) webhook asks for security attacks.
I interpreted "Verifies that it can remove the broken webhook" as just ensuring that no webhook can be registered which could potentially deny its own (or another webhook in the same namespace's) deletion. Broken webhooks definitely shouldn't be treated differently in terms of determining who can delete them, but validation should probably prevent webhooks from applying to creates, updates and deletes of webhook objects, wildcard selectors should exclude webhook objects, etc. Also, this behavior would need to be documented. I think the safest way to ensure that a webhook does not get deleted is with RBAC rules, rather than using a validating webhook to prevent its deletion. I can't think of a legitimate use case for webhooks which affect the admission of webhook objects, it seems like any workflow which relies on that could be implemented in another, easier to reason about way. thoughts?
I have the same understanding. Excluding webhooks from webhooks seems reasonable. But it's only one piece in the (restart when webhook broke) puzzle. We reconcile RBAC rules, create namespaces (or keep them when they exist already). So I expect some more exceptions.
Is this still something you are working on?
If you can make a webhook which prevents the apiserver from starting at all, this is a problem, no matter what the webhook is verifying/doing, because you can't recover from that state.

I'm not suggesting that the system should automatically remove a broken webhook, which would be a security problem, rather that it needs to be possible for an appropriately authenticated sysadmin to do so. Especially since making a broken webhook non-broken may require executing API calls that the broken webhook is preventing.
/assign
as you assigned this to you, can you sketch what the plan is for this?
The plan was to use an integration test, (i.e. ) which I implemented in but it turns out that since the integration test apiserver does not follow exactly the same startup chain as the real one, that it is insufficient to ensure the apiserver can actually come up with a broken webhook installed. The integration test doesn't even fail against the code before merged. So currently the plan is to do it as an e2e test, probably with the [Slow] and [Destructive] tags.
it should be 99.9% the same. What are you missing? I think an integration test would be much better than an e2e test. We have much more control there.
I think the bug fixed was something a controller (bootstrap controller?) was doing, not the apiserver
those bootstrap controller are also executed by the testserver. So I would expect that the bug could be hit in testserver integration tests too. Compare with the real start code and for the test server start code. They are the same. The only difference is that we tweak the options beforehand a bit to run with a random port and random etcd prefix.
Hmm, then I think the reason that the test doesn't fail pre- is because the post start hook changed by that PR isn't called unless api-aggregation is being used. I guess the integration test for this will have to do that as well. Although the one in is still somewhat valuable to make sure new startup namespace creations in non aggregated apiservers are done correctly.
- you probably already thought about it, but one more thing to ensure here is that: apiserver can starte responding "OK" to "/healthz" even if all webhooks are broken [that means e.g. finishing all post-startup hooks] Without that, apiserver would be consistently being killed + restarted by Kubelet on which it is running (at least in some setups).

Yeah, the post-start hooks are the thing that inspired this issue :) Wojciech Tyczynski <:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen : — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: goodSum += histogram[count] } <del> goodPct := 100 * float64(goodSum) / float64(allCoordinateCount) </del> <ins> goodPct := 100 * float64(goodSum) / float64(numHandsSeen) </ins> t.Logf("good percentage = %v, histogram = %v", goodPct, histogram) <del> if goodSum != allCoordinateCount { </del> <ins> if goodSum != numHandsSeen { </ins> t.Errorf("Only %v percent of the hands got a central count", goodPct) } }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which test(s) are flaking: Testgrid link: While there aren't any visible failures in the above testgrid link, we've observed flaky timeouts downstream. Reason for failure: Test times out Anything else we need to know: We don't have full logs for failed unit tests, so it's hard to pinpoint exactly which test is timing out, but I believe the offending function is this: You will see that the test iterates times: Summing up the hashMax for all four subtests, we see that the function iterates ~34 million times. While the test takes just ~9 seconds on my machine to complete, the CI is probably struggling with it (other tests are running in parallel and iterating 34 million times over a non-trivial loop is very CPU hungry). A successful execution of the unit test on CI takes ~3-4 minutes on average. The test doesn't have any of the usual suspects that can cause a timeout (like <-chan receive, i/o reads, wait.Poll(), time.Sleep(), etc.), so I suspect the large number of iterations to be the culprit. I propose making the unit test smaller and quicker to complete. Hopefully someone who understands shufflesharding better than me can take over :) /sig api-machinery /cc
when run with , the runtime increases significantly: that time is dominated by TestUniformDistribution
The test does a lot of slice appends. Maybe is constraining the allocator? There isn't much else in the test that might be slowed down by : no mutexes, no RNGs, no atomic operations, no channels, etc.
It looks like the buffer is pre-allocated to be the correct size though, so none of those appends should be resulting in allocations?
/triage accepted
BTW, in another branch I tried taking the s out of the test, instead calling with a lambda that writes into at a position selected by an advancing index. The got a little slower!


Question: ns := "" testSimpleCRUD(t, ns, noxuDefinition, noxuVersionClient) <ins> testFieldSelector(t, ns, noxuDefinition, noxuVersionClient) </ins> } func testSimpleCRUD(t *testing.T, ns string, noxuDefinition *apiextensionsv1beta1.CustomResourceDefinition, noxuVersionClient dynamic.Interface) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: got an error when trying to use a fieldSelector when watching a custom resource What you expected to happen: no error How to reproduce it (as minimally and precisely as possible): a CRD and then a CR for it. to watch with a fieldSelector, such as Anything else we need to know?: The customresource implementation has a defined, and it looks like it should support at least a field selector for and , but I'm guessing we need a bit more in place to make this work for . Environment: Kubernetes version (use ): HEAD (commit ) /sig api-machinery I can try to submit a PR to fix once I finish my current work item, if nobody picks this up sooner.
/area third-party-resource
cc
This is a more general form of . I don't have immediate plans to work on either of them, so you're welcome to take it.
sounds good. Do you want me to close this as a dupe?
I think there's a useful distinction between the two issues. If we support only , that would be enough to but this one tracks the idea of supporting arbitrary field selectors. By the way, if I recall correctly, when I applied the hack mentioned in the other thread to skip the step for CRDs, I think I saw that actually arbitrary field selectors worked too (not just ). That would mean enabling this might be easy on a technical level, although we may need to think through whether this could tie our hands if we ever wanted to add versioning support for CRD in the future. This issue is a better home for that discussion than the other one.
This would help a lot with debugging in prow. Today, the only thing we can do is bubble up information from the spec of a prowjob as labels and use a label selector.
What about just having 's just return the passed in label and value without any "conversion" for now?
I don't think we'd want to do it for all Unstructured, since there's no guarantee all Unstructured objects are CRD objects. However, it should be pretty easy to arrange for ConvertFieldLabel to be a no-op for CRD objects, which makes sense currently because CRD only allows one version for a given resource.

We just need to make sure this will not prevent us from doing more with CRD versioning in the future. My suspicion is that it should be fine, but I haven't investigated properly.
ok, I can put together a PR to make this change. any gut reactions?
PR:
I am sorry to comment on this closed thread. Currently, I am working on a CRD and needs to support arbitrary field selectors. I've tried to do it by learning from what you do for . But it always throws error: . So, I am wondering if I am doing something wrong or it only supports limited field selectors for CRD, which is stated . Here is my code snippet:
CRDs still only support namespace and name for field selectors
Any plan to support arbitrary field selector for CRDs?
Not currently

Follow a TPR an instance a single instance return 400. (watch a collection of TPR works) master and kubectl version: 1.7.1 CRD has the same issue. /assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I was able to reproduce this with CRD on master. The problem is that kubectl implements a watch on a single object as a watch on the collection with a field selector that filters by . However, field selectors don't work on CRD objects because they are stored as , which does not support : This method is used to transform the field selector from versioned to internal field names: As a test, I bypassed that check for Unstructured objects, and watches on single CRD objects started working. Given that CRD doesn't support multiple versions anyway, it seems feasible to support this with a properly-targeted override of specific to CRD objects. cc
If we could support name and namespace generally on all objects containing object meta, I think I'm in favor of the concept.
I think it's feasible to fix this for CRD at some point. However, given that TPR is already gone in master, I don't think it's worth fixing for TPR unless it's somehow a regression in 1.7. Do you have any objections to that?
Having CRD support is good enough for me and service catalog. Will you backport the fix for CRD to 1.7?
At this stage I wouldn't expect that since it isn't a regression (I don't think) for either extension mechanism.
cc
I am also getting the same error when i try to watch CRDs and filter on a specific field. Im a currently using the to watch for CRDs. On a related note, i noticed that the takes a . Is there a way to pass a label selector so that i can watch CRDs using labels?
Also is this issue slated for 1.8?

(For CRDs that is)
The general issue of supporting field selectors for CRD is tracked in . I don't think it's slated for 1.8. The methods on should take a , which can contain an optional .


Question: "name":       "The name of this port within the service. This must be a DNS_LABEL. All ports within a ServiceSpec must have unique names. This maps to the 'Name' field in EndpointPort objects. Optional if only one ServicePort is defined on this service.", "protocol":   "The IP protocol for this port. Supports "TCP" and "UDP". Default is TCP.", "port":       "The port that will be exposed by this service.", <del> "targetPort": "Number or name of the port to access on the pods targeted by the service. Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. If this is a string, it will be looked up as a named port in the target Pod's container ports. If this is not specified, the value of Port is used (an identity map). Defaults to the service port. More info: http://releases.k8s.io/HEAD/docs/user-guide/services.md#defining-a-service", </del> <ins> "targetPort": "Number or name of the port to access on the pods targeted by the service. Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. If this is a string, it will be looked up as a named port in the target Pod's container ports. If this is not specified, the value of the 'port' field is used (an identity map). This field is ignored for services with clusterIP=None, and should be omitted or set equal to the 'port' field. More info: http://releases.k8s.io/HEAD/docs/user-guide/services.md#defining-a-service", </ins> "nodePort":   "The port on each node on which this service is exposed when type=NodePort or LoadBalancer. Usually assigned by the system. If specified, it will be allocated to the service if unused or else creation of the service will fail. Default is to auto-allocate a port if the ServiceType of this Service requires one. More info: http://releases.k8s.io/HEAD/docs/user-guide/services.md#type--nodeport", }

In 1.0 and 1.1, headless services that specified a mismatched port/targetPort were tolerated (the targetPort was ignored). tightened validation to reject these services. That causes problems when upgrading from currently-working clusters.
2 candidate to avoid breaking backwards compatibility
oh hell. Yeah. I don't think this is worth a breaking change. We should either roll that back ASAP or change it to nuke any user-provided value and replace it with the value. I think the latter is potentially dangerous and also not worth the risk, so I think we should rollback and just update the docs for targetPort. objections?
Who has time to do this? It doesn't look like an auto-rollback is doable.
Already opened a PR in
Yes, I agree this isn't worth a breaking change, But could we give an warning log that targetPort isn't equal to port and would be ignored?
There's no facility in validation for warning. I don't think spamming the server log in response to API requests is that helpful here
Yes, that sounds reasonable. I just think that targetPort was ignored is a little unfriendly to user.


Question: // actions will only have sandbox ID and not have pod namespace and name information. // Return error if encounter any unexpected error. if checkpointErr != nil { <ins> if checkpointErr != errors.ErrCheckpointNotFound { err := ds.checkpointManager.RemoveCheckpoint(podSandboxID) if err != nil { glog.Errorf("Failed to delete corrupt checkpoint for sandbox %q: %v", podSandboxID, err) } } </ins> if libdocker.IsContainerNotFoundError(statusErr) { glog.Warningf("Both sandbox container and checkpoint for id %q could not be found. "+ "Proceed without further sandbox information.", podSandboxID)

/kind bug are failing. seems like the likely cause. cc /assign
ping
seems one test still failing.


Question: deps = [ "//pkg/api/testapi:go_default_library", "//pkg/api/v1:go_default_library", <del> "//pkg/apis/extensions:go_default_library", </del> <ins> "//pkg/apis/extensions/v1beta1:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", ], )

Here are the hash label keys of all controllers: Deployment: DaemonSet: (new in 1.7) StatefulSet: (new in 1.7) Had a brief discussion with and about this, it should be consistent. We can use for DaemonSet and StatefulSet, and keep Deployment as-is. Deployment hash label key should be different from other two because Deployment pods hash labels is used by selector (unlike the other two controllers). We probably should have used hash annotations instead of hash labels for DaemonSet and StatefulSet, since we don't use hash labels to select their history or pods. Hash labels are chosen for consistency. This is fine since we don't modify hash label values.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Actually, controller revisions created by statefulset, daemonset has different label: // controller revision created by statefulset label: "" // controller revision created by daemonset label: controller-revision-hash: "" should be consistent?


Question: }, "dataSource": { "$ref": "#/definitions/io.k8s.api.core.v1.TypedLocalObjectReference", <del> "description": "This field requires the VolumeSnapshotDataSource alpha feature gate to be enabled and currently VolumeSnapshot is the only supported data source. If the provisioner can support VolumeSnapshot data source, it will create a new volume and data will be restored to the volume at the same time. If the provisioner does not support VolumeSnapshot data source, volume will not be created and the failure will be reported as an event. In the future, we plan to support more data source types and the behavior of the provisioner may change." </del> <ins> "description": "This field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot - Beta) * An existing PVC (PersistentVolumeClaim) In order to use VolumeSnapshot object types, the appropriate feature gate must be enabled (VolumeSnapshotDataSource) If the provisioner can support the specified data source, it will create a new volume based on the contents of the specified PVC or Snapshot. If the provisioner does not support the specified data source, the volume will not be created and the failure will be reported as an event. In the future, we plan to support more data source types and the behavior of the provisioner may change." </ins> }, "resources": { "$ref": "#/definitions/io.k8s.api.core.v1.ResourceRequirements",

/kind feature /sig storage /sig api PVC Data Sources have been in beta since 1.16 and should be ready to be promoted to GA in the 1.18 release. KEP:
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /kind feature /sig storage /sig api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: switch { case err == nil: // Pod was created, feature supported. <del> StopPod(c, pod) </del> <ins> StopPodAndDependents(c, pod) </ins> return true, nil case apierrors.IsInvalid(err): // "Invalid" because it uses a feature that isn't supported.

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: I0722 06:26:18.056] I0722 06:26:18.057] [Driver: csi-hostpath] I0722 06:26:18.057] I0722 06:26:18.057] [Testpattern: Generic Ephemeral-volume (default fs) [Feature:GenericEphemeralVolume] (late-binding)] ephemeral skipped 448 lines unfoldmore I0722 06:30:33.845] Jul 22 06:30:33.845: INFO: Wait up to 5m0s for pod "inline-volume-tester2-hhcbd" to be fully deleted I0722 06:31:06.117] Jul 22 06:31:06.117: INFO: Pod inline-volume-tester-74jqh has the following logs: /dev/sda1 on /mnt/test-0 type ext4 (rw,nosuid,nodev,noexec,relatime,commit=30) I0722 06:31:06.117] I0722 06:31:06.117] Jul 22 06:31:06.117: INFO: Deleting pod "inline-volume-tester-74jqh" in namespace "ephemeral-9141" I0722 06:31:06.155] Jul 22 06:31:06.155: INFO: Wait up to 5m0s for pod "inline-volume-tester-74jqh" to be fully deleted I0722 06:31:38.264] Jul 22 06:31:38.264: FAIL: no dangling PVCs I0722 06:31:38.264] Expected I0722 06:31:38.265] <[]v1.PersistentVolumeClaim | len:1, cap:1: [ I0722 06:31:38.265] { I0722 06:31:38.265] TypeMeta: {Kind: "", APIVersion: ""}, I0722 06:31:38.

265] ObjectMeta: { skipped 315 lines unfold_more I0722 06:32:16.896] STEP: Destroying namespace "ephemeral-9141" for this suite. I0722 06:32:16.931] Jul 22 06:32:16.931: INFO: Namespace ephemeral-9141 was already deleted I0722 06:32:16.932] STEP: Destroying namespace "ephemeral-522" for this suite. I0722 06:32:16.966] Jul 22 06:32:16.966: INFO: Namespace ephemeral-522 was already del Anything else we need to know: /cc /sig storage /priority critical-urgent /milestone v1.19
I have tracked this down and it looks like that changes were made in that affected how Pods were being deleted in the e2e framework (specifically commit ). This caused failures in other tests, so the commit was reverted in I will have a fix open shortly that uses foreground deletion only for these storage tests. /assign /cc Side note: I see already saw this


Question: readonly KUBE_CLIENT_PLATFORMS elif [[ "${KUBE_FASTBUILD:-}" == "true" ]]; then <del> KUBE_SERVER_PLATFORMS=(linux/amd64) </del> <ins> host_arch=$(kube::util::host_arch) if [[ "${host_arch}" != "amd64" && "${host_arch}" != "arm64" ]]; then # on any platform other than amd64 and arm64, we just default to amd64 host_arch="amd64" fi KUBE_SERVER_PLATFORMS=("linux/${host_arch}") </ins> readonly KUBE_SERVER_PLATFORMS <del> KUBE_NODE_PLATFORMS=(linux/amd64) </del> <ins> KUBE_NODE_PLATFORMS=("linux/${host_arch}") </ins> readonly KUBE_NODE_PLATFORMS if [[ "${KUBE_BUILDER_OS:-}" == "darwin"* ]]; then KUBE_TEST_PLATFORMS=( <del> darwin/amd64 linux/amd64 </del> <ins> "darwin/${host_arch}" "linux/${host_arch}" </ins> ) readonly KUBE_TEST_PLATFORMS KUBE_CLIENT_PLATFORMS=( <del> darwin/amd64 linux/amd64 </del> <ins> "darwin/${host_arch}" "linux/${host_arch}" </ins> ) readonly KUBE_CLIENT_PLATFORMS else <del> KUBE_TEST_PLATFORMS=(linux/amd64) </del> <ins> KUBE_TEST_PLATFORMS=("linux/${host_arch}") </ins> readonly KUBE_TEST_PLATFORMS <del> KUBE_CLIENT_PLATFORMS=(linux/amd64) </del> <ins> KUBE_CLIENT_PLATFORMS=("linux/${host_arch}") </ins> readonly KUBE_CLIENT_PLATFORMS fi else

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When trying to build kubectl from tip () on a M1 Macbook: Why is this needed: M1 MacBooks are a new major client platform that many developers will use. brew listed kubectl (kubernetes-cli) is one of the major packages still not supported on M1 () Since it's supported by go, it's only required to add it to the supported client list Locally I managed to add support using the following patch:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli architecture
/sig release
Any updates on this? cheers
you can watch the PR:
it works, nice!!!
Done!!!


Question: frameworkOutOfTreeRegistry framework.Registry profiles                   []schedulerapi.KubeSchedulerProfile extenders                  []schedulerapi.Extender <ins> frameworkCapturer          FrameworkCapturer </ins> } // Option configures a Scheduler

has compatibility tests for Policy v1. We are lacking similar tests for other options of , such as: [x] : This can be right away to test that default plugins can be disabled, changed order, etc. [x] : This one is trickier, as it requires a new method in to expose all configurations and a new method in so that they can say which args they used. /sig scheduling /help /kind cleanup
/assign
you can work on the first one. I'll have a PR soon that should clarify how the second one should be done.
I have sent a pr for the first one. And I'm working on the second one, I think it's better to keep 2 PRs for this issue? If so, PTAL at the first one, thanks.
Yes, 2 PRs is ideal. I'll review later today.
I have one small question about do I need to mock some fake plugins for testing , or just testing the default plugins?
You should use the existing plugins, because these are compatibility tests.
Okay
Actually, I think adding a new method such as to interface is a little redundant. In fact, most plugins don't need any argument for building them, so we are talking about adding a new method that needs to be implemented by all of the existing and coming plugins. And users who write out-of-tree plugins also hardly have the need for using this method. IMO, it would be more meaningful to move up to the interface. Then by this method, we can simply implement for default plugins who actually need args. WDYT?
should remain unexported because it provides pointers. However, I agree that might be too much to ask for all plugins. Instead, we can add a new interface and then plugins can opt-int. Not sure if we should follow this extension pattern Any thoughts
On the other hand, simply returning doesn't seem like much.
Sure, then I will add like what we want before(add this method to interface), and if there is any better idea later, please let me know.
I think we should just add it to the plugins that actually have args, this will require custom code in the test, but I think it is better than having all plugins implement a function means something to only 1% of them.
Thanks, but now there is a problem that we can't get the registered plugins with current API.

So, I'm not sure if we can just implement the method in with some reflect code for checking if one plugin has implement method? This could help us with keeping the interface clean. cc
you don't need to check if it implements BuildArgs, since this is for testing, we can hardcode which plugins should have their args verified.
The test will not be pretty, but I think it is worth the tradeoff.
Do you mean that we don't test the default plugins, but define some custom plugins in the test file? And we don't need the ListPluginArgs method anymore, am I right?
and I discussed this offline and arrived to this conclusion: For the compatiblity test, we can mock ( in ) to capture the arguments the configuration that is being passed. This can also be used by with We want another test for that checks the build args for the in-tree plugins accessing the internal pointers. These 2 tests should cover the whole stack. Sorry for the back-and-forth. Sometimes it's useful to see a full PR to realize that the initial idea wasn't good enough. Thanks a lot for it.
Sure, that sounds much more clear, thanks. :D
One more question, the is under package , and the compatibility test is under package . So for mocking , we need to change to a function variable in , and add a method for it, right It will look like: Maybe we can add something like to , but that's the implementation details. I want to make sure if the above design is proper.
Or rather, you could add something like: . Then you can call the capturer from if there is one defined.
Got your message in slack, let me reply here. sounds good, and do I need to revert the change in ? I think is not necessary now.
I think they will still be useful for the other test:
Hello are you still working on this?
Ah, I'm so sorry that I'm so busy in the last few weeks, but I will send prs fot this issue in this weekend. Sorry for my delay again.


Question: if _, err := fmt.Fprintf(w, "%st%vt%vt%vt%s", name, <del> "-", backendStringer(ingress.Spec.Backend), </del> <ins> formatHosts(ingress.Spec.Rules), </ins> loadBalancerStatusStringer(ingress.Status.LoadBalancer), <ins> formatPorts(ingress.Spec.TLS), </ins> translateTimestamp(ingress.CreationTimestamp), ); err != nil { return err

We should show a nicely formatted map, with tls and ports. This 50% works. Today we have: Which works well enough, but printing a single Ingress involves hand managing tabs and newlines till the entire Ingress is printed, i.e: So when we start mixing different Ingress types, we get jarring output: This is because the columns are formatted per the top ingress and don't match the bottom one. The tabbed writer used by kubectl to align columns gets confused on the whole.
you can assign this to me, will fix this next-candidate.
thanks!
Sounds like we should keep it to a single line () and move everything else to describe. Makes this bug easier.
yeah, we will move this to a single line, this is the principle.
was this finished for 1.3?
the code is already finished on my local branch. I will open a PR as I confirmed it.
what do you think if print ingress like this: Also the original resource is:
That would fall right off the screen. I think we have to stick to just printing the ips in , and everything else in . You can print hoststs alone, and optimize for small ingresses by doing something like we do for endpoints (show a small number of characters then do "..."). The problem with trying to print everything on one line is you can't represent the fanout cleanly, ie you might do: But with multiple hosts: Doesn't work, because /bar and secret2 only applies to
I agree with this, maybe we should just remove these rules from get ingress output or just leave host here.
SGTM are you going to send a pr?
yes, ready to.


Question: JSONPatchMaxCopyBytes:       -10 * 1024 * 1024, MaxRequestBodyBytes:         10 * 1024 * 1024, }, <del> expectErr: "--json-patch-max-copy-bytes can not be negative value", </del> <ins> expectErr: "ServerRunOptions.JSONPatchMaxCopyBytes can not be negative value", </ins> }, { name: "Test when MaxRequestBodyBytes is negative value",

kube-apiserver has no --max-resource-write-bytes & --json-patch-max-copy-bytes option. Correct the error message to not use the "--max-resource-write-bytes" & "--json-patch-max-copy-bytes" string. Check the source code: No response v1.19 ~ v1.23 eg: </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /area apiserver /kind cleanup /assign FYI:
/cc /triage accepted


Question: <ins> package etcd import ( "errors" "net" "time" ) // dial attempts to open a TCP connection to the provided address, explicitly // enabling keep-alives with a one-second interval. func Dial(network, addr string) (net.Conn, error) { conn, err := net.DialTimeout(network, addr, time.Second) if err != nil { return nil, err } tcpConn, ok := conn.(*net.TCPConn) if !ok { return nil, errors.New("Failed type-assertion of net.Conn as *net.TCPConn") } // Keep TCP alive to check whether or not the remote machine is down if err = tcpConn.SetKeepAlive(true); err != nil { return nil, err } if err = tcpConn.SetKeepAlivePeriod(time.Second); err != nil { return nil, err } return tcpConn, nil } </ins>

When evaluating results of I have noticed a very strange behavior of apiserver when running density tests on a 100 node cluster with 30 pods per node. It seems that sometimes we spent more than 40% of CPU time in net.(*netFD).dial function. My suspicion is that it's because after each watch message we terminate old connection to etcd and create a new one. With multiple watchers and many update this can be very expensive.
!
Does the etcd client not reuse HTTP connections? I thought the go http client did that by default?
Actually looking at that chart I see it stuck in write, not dial-- am I blind?
I think it is write... The number on dail is confusing though...
There is a small, but very heavy branch on the left of the chart with "dial" path".
But it shows . It does not show where is the other 22.87s.
The way I read this profile (might wrong though) is that for 28.90s dial was in the stack, but was not the "top" function for 22.87s I tried to verify that etcd client reuses HTTP connection and I think it doesn't. I simple log line here: In empty 100 node cluster it got called ~10k times in ~3.5 min. This gives 45 dials per second Do you have a theory why we create so many connections? After adding cache in this is responsible for a biggest chunk of CPU time.
I also run density test to check how correlated this is with the load.

Results below (first column is number of dials per minute, second column is time): $ sudo docker logs 2&1 uniq -c 1839 13:12 4737 13:13 1942 13:14 1184 13:15 836 13:16 949 13:17 1360 13:18 880 13:19 770 13:20 1127 13:21 682 13:22 13:23 13:24 13:25 13:26 13:27 13:28 13:29 13:30 13:31 13:32 13:33 13:34 13:35 13:36 13:37 13:38 13:39 13:40 13:41 13:42 13:43 13:44 1314 13:45 244 13:46 Performance test started at 13:23 and finished around 13:44. Correlation is quite obvious. Note also that in the peak it was generating 560 dials per second!
I think that we have a victim for it. What we've done is that we changed the Transport in go-etcd client: but setting: MaxIdleConnsPerHost: (from default 2) This speeded up the test by ~35% I think that is some exaggeration, but it definitely proves that increasing this value solves the problem.

My hypothesis why it solves the problem: have 100 opened watches( watching for the same changes (which definitely happens in 100-node cluster) and all of them are from apiserver (a single client) change that is interesting is written to etcd is responding to all of them at the same time those connections become iddle (or most of them) before the new request is send, and thus most of them are closed (we can have only 2 of them opened for a single client) the next requests within a watch we need to create new connections
Number of dials after change described by :+1: ~$ sudo docker logs 2&1 uniq -c 335 14:29 200 14:30 5 14:31 28 14:32 65 14:33 145 14:34 69 14:35 121 14:36 59 14:37 202 14:39 140 14:40 112 14:42 77 14:43 190 14:44 104 14:45 5 14:46 4 14:47 (this includes run of performance test)
Just a silly question: are we allowed to change files under Godeps/ directory? If not what is the best way to fix it?
Call instead.
is probably right. If you only use one transport, there is a maxIdelConn limitation.
thanks ! Assigning to myself - I will prepare a PR for it.
cool, this is great. max idle is probably a little high-- we don't want to use all of apiserver's available FDs for this. Does, say, 500 still get most of the benefits? (I think if we multiplexed watches that might also help a lot, but obviously it's much easier to keep more idle connections.) (If you really had needed to modify a file in the godeps/ directory, the proper way to do that is to send a PR to the downstream repo, and the update the dependency once they accept it.)
I completely agree is too much. I didn't have time to make more experiments today, but my feeling is that 500 or 1000 should also be enough. I will come up with more accurate number on Monday. Yes - but this would be much bigger change.

I don't think it's worth doing now. It seems it's ot needed. We just need to initialize the client properly. I will create a PR for it.


Question: } } <del> if !skipPreFlight { fmt.Println("Running pre-flight checks") err := preflight.RunInitMasterChecks(cfg) if err != nil { return nil, &preflight.PreFlightError{Msg: err.Error()} } } else { fmt.Println("Skipping pre-flight checks") } </del> // Auto-detect the IP if len(cfg.API.AdvertiseAddresses) == 0 { // TODO(phase1+) perhaps we could actually grab eth0 and eth1

Environment: VMware CentOS Linux release 7.2.1511 (Core) kernel: 3.10.0-327.36.1.el7.x8664 Installed packages: kubeadm.x8664 1.5.0-0.alpha.0.1534.gcf7301f kubectl.x8664 1.4.0-0 kubelet.x8664 1.4.0-0 kubernetes-cni.x8664 0.3.0.1-0. docker-engine.x8664 1.11.2- docker-engine- 1.12.1- What happened: kubeadm init hangs indefintely after: kubeadm init <master/tokensgenerated token: "." <master/pkicreated keys and certificates in "/etc/kubernetes/pki" <util/kubeconfigcreated "" <util/kubeconfigcreated "" <master/apiclientcreated API client configuration <master/apiclientcreated API client, waiting for the control plane to become readykubeadm init What you expected to happen: Master node initialize Anything else do we need to know: Only the kubeadm and kublet processes are running. When I do docker ps no containers are running. If i type docker images the only image that has been downloaded are: I've also tried: kubeadm init --api-advertise-addresses but with the same outcome (followed the clean-up at ) Im behind a firewall so I'm using proxy settings for docker. I have confirmed I can reach images on docker hub and google cloud registry
will download the required binaries viz etcd. When you are behind firewall kubeadm will hang/wait forever to download the binaries. I dont know whether it is addressed correctly in kubeadm implementation. I am also facing the same issue and waiting for the resolution.
Thx, I suspected it to be some firewall issue. Do you know if there's a way to add httpsproxy & httpproxy settings to kubeadm?
Kubeadm doesn't seem to pick proxy environment variables. I am looking at recompiling with proxy support may work. But I dont know how to do that. Waiting for updates about proxy
This issue same as Lets join there. The solution mentioned there may work.

You can try kubeadm with patch applied from PR : It works at least in my environment behind restrictive proxy.
thanks, I will look into it


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/runtime" </ins> genericapirequest "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/registry/generic" genericregistrytest "k8s.io/apiserver/pkg/registry/generic/testing" "k8s.io/apiserver/pkg/registry/rest" etcdtesting "k8s.io/apiserver/pkg/storage/etcd/testing" api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/registry/registrytest" )

What happened: I created a namespace with finalizer, and then tried to delete it: The namespace can be deleted, and the finalizer is removed by garbage collection controller: What you expected to happen: Since kubectl species , it is not an orphaning operation, the finalizer should be removed in namespace store. How to reproduce it (as minimally and precisely as possible): Same as what I observed: a namespace. finalizer. . Anything else we need to know?: Environment: Kubernetes version (use ): HEAD
/sig api-machinery
cc
Discussion on slack:


Question: package options import ( <ins> "net" </ins> utilnet "k8s.io/apimachinery/pkg/util/net" ) // DefaultServiceNodePortRange is the default port range for NodePort services. var DefaultServiceNodePortRange = utilnet.PortRange{Base: 30000, Size: 2768} <ins> // DefaultServiceIPCIDR is a CIDR notation of IP range from which to allocate service cluster IPs var DefaultServiceIPCIDR net.IPNet = net.IPNet{IP: net.ParseIP("10.0.0.0"), Mask: net.CIDRMask(24, 32)} </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: There is currently no easy way to know what Kubernetes is using as the default CIDR for service cluster IPs. What you expected to happen: should be able to tell users that "10.0.0.0/24" is used as the default. They'd better check if their deployment environment has a conflict with this assumption. They need to customize this parameter to avoid such overlaps if any. How to reproduce it (as minimally and precisely as possible): Run , from the help, the default value for --service-cluster-ip-range is not mentioned. Anything else we need to know?: No. Environment: Kubernetes version (use ): Any Cloud provider or hardware configuration**: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: N/A Others: N/A
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale


Question: }) existingVMForAS1 := buildDefaultTestVirtualMachine("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/availabilitySets/agentpool1-availabilitySet-00000000", []string{"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-1"}) existingVMForAS2 := buildDefaultTestVirtualMachine("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/availabilitySets/agentpool2-availabilitySet-00000000", []string{"/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool2-00000000-nic-1"}) <del> existingNIC := buildDefaultTestInterface(true, []string{"/subscriptions/sub/resourceGroups/gh/providers/Microsoft.Network/loadBalancers/testCluster/backendAddressPools/testCluster"}) </del> <ins> existingNICForAS1 := buildDefaultTestInterface(true, []string{"/subscriptions/sub/resourceGroups/gh/providers/Microsoft.Network/loadBalancers/testCluster/backendAddressPools/testCluster"}) existingNICForAS1.VirtualMachine = &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-00000000-1"), } existingNICForAS2 := buildDefaultTestInterface(true, []string{"/subscriptions/sub/resourceGroups/gh/providers/Microsoft.Network/loadBalancers/testCluster/backendAddressPools/testCluster"}) existingNICForAS2.VirtualMachine = &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Compute/virtualMachines/k8s-agentpool2-00000000-1"), } </ins> mockVMClient := mockvmclient.NewMockInterface(ctrl) mockVMClient.EXPECT().Get(gomock.Any(), cloud.ResourceGroup, "k8s-agentpool1-00000000-1", gomock.Any()).Return(existingVMForAS1, nil) mockVMClient.EXPECT().Get(gomock.Any(), cloud.ResourceGroup, "k8s-agentpool2-00000000-1", gomock.Any()).Return(existingVMForAS2, nil) cloud.VirtualMachinesClient = mockVMClient mockNICClient := mockinterfaceclient.NewMockInterface(ctrl) <del> mockNICClient.EXPECT().Get(gomock.Any(), "rg", "k8s-agentpool2-00000000-nic-1", gomock.Any()).Return(existingNIC, nil) </del> <ins> mockNICClient.EXPECT().Get(gomock.Any(), "rg", "k8s-agentpool1-00000000-nic-1", gomock.Any()).Return(existingNICForAS1, nil) mockNICClient.EXPECT().Get(gomock.Any(), "rg", "k8s-agentpool2-00000000-nic-1", gomock.Any()).Return(existingNICForAS2, nil).Times(3) </ins> mockNICClient.EXPECT().CreateOrUpdate(gomock.Any(), gomock.Any(), gomock.Any(), gomock.Any()).Return(nil) cloud.InterfacesClient = mockNICClient cleanedLB, err := cloud.cleanBackendpoolForPrimarySLB(&lb, &service, clusterName)

What happened: introduced new dependency on Availability Sets which is too restrictive for some existing installations: What you expected to happen: Azure cloud provider to successfully manage LBs, like it had been doing before. Instead, it died [with][1]: How to reproduce it (as minimally and precisely as possible): Fails every time in OpenShift CI. Can you point us to documentation around the infrastructure options that the Azure cloud provider expects for LB management? https://kubernetes- doesn't say anything about NIC names, and it doesn't have specifics around what things should look like without Availability Sets, or whether "without Availability Sets" is not supported (when previous iterations of the cloud provider had no problem with our lack of Availability Sets). Documenting cloud-provider assumptions about host hardware would have also helped avoid ; CC /sig cloud-provider /area provider/azure [1]:
/triage accepted
/assign
Notes on OpenShift's Azure infrastructure in ARM templates [here][1], in case that helps clarify the infra we use today. Those are OpenShift 4.6 docs, and 4.6 is based on Kubernetes 1.19, and things worked fine. The trouble we've been having when we took the same Azure-infra approach and tried to use it with Kubernetes 1.20, after openshift/kubernetes. [1]:
could you share an example of resourceID (please replace your subscription/resourceName with fake ones)?
Example CI run [here][1] from the NIC-renaming openshift/installer. Installer logs [here][2] include: I'm hoping that the subscription name is not sensitive, because we've been dumping them in public logs for a long time now . Should be IDs for any other resources you're interested in in those logs as well. [1]: [2]: https://gcsweb-
Looks like this is about the master node. Are you using standard LB or a basic one? For standard LB, the master node would be excluded from the LB by default.
[Standard][1] (and [here][2]). Is this documented somewhere? We've been using standard LBs [for years][3], with no problems until the bump to the Kube 1.20 Azure cloud-provider. [1]: [2]: [3]:
Are you referring to the built-in logic of Kube to never target master nodes with a service type LoadBalancer?

That used to be a problem (), but my understanding is that it has since been fixed.
Can we re-open this? kubernetes-sigs/cloud-provider-azure was [not sufficient][1] to get the cloud-provider happy on our infra, which again, contains no Availability Sets. There are some more notes about our infrastructure [here][2] (with discussion in [rhbz][3]), in case that helps. [1]: [2]: [3]:
who has a better grasp on this than me, also just opened . Sounds like fixing that broader issue might obsolete this ticket, because "the cloud-provider expects Availability Sets which OpenShift infra doesn't have in order to figure out which nodes to remove" doesn't matter if the cloud-provider isn't removing any OpenShift nodes in the first place.
can you please confirm the standalone VM scenario is supported? It's a critical use case for CAPZ (our control plane VMs in zone-enabled regions are not placed in availability sets)

What happened: introduced new regex which is too restrictive for some existing installations: This new restriction causes LBs to fail to sync if just one ipConf fails regex. What you expected to happen: This change was to enable optional behavior, any ipConf that fails regex should just be ignored. This will allow existing installs to work as they did prior to the change. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Suggest putting a continue here: This will restore previous functionality and appears to have no undesirable impact.
/sig cloud-provider /area provider/azure
Unclear if we should also put a continue here as well:
/assign
could you share an example of resourceID (please replace your subscription/resourceName with fake ones)? /triage accepted
Here's the error we're seeing: Just the string


Question: func (az *Cloud) isCurrentInstance(name types.NodeName) (bool, error) { nodeName := mapNodeNameToVMName(name) metadataName, err := az.metadata.Text("instance/compute/name") <ins> if err != nil { return false, err } if az.VMType == vmTypeVMSS { // VMSS vmName is not same with hostname, use hostname instead. metadataName, err = os.Hostname() if err != nil { return false, err } } metadataName = strings.ToLower(metadataName) </ins> return (metadataName == nodeName), err } // InstanceID returns the cloud provider ID of the specified instance. // Note that if the instance does not exist or is no longer running, we must return ("", cloudprovider.InstanceNotFound) func (az *Cloud) InstanceID(ctx context.Context, name types.NodeName) (string, error) { <ins> nodeName := mapNodeNameToVMName(name) </ins> if az.UseInstanceMetadata { isLocalInstance, err := az.isCurrentInstance(name) if err != nil { return "", err } <del> if isLocalInstance { nodeName := mapNodeNameToVMName(name) return az.getMachineID(nodeName), nil </del> <ins> // Not local instance, get instanceID from Azure ARM API. if !isLocalInstance { return az.vmSet.GetInstanceIDByNodeName(nodeName) } // Compose instanceID based on nodeName for standard instance. if az.VMType == vmTypeStandard { return az.getStandardMachineID(nodeName), nil } // Get scale set name and instanceID from vmName for vmss. metadataName, err := az.metadata.Text("instance/compute/name") if err != nil { return "", err } ssName, instanceID, err := extractVmssVMName(metadataName) if err != nil { return "", err </ins> } <ins> // Compose instanceID based on ssName and instanceID for vmss instance. return az.getVmssMachineID(ssName, instanceID), nil </ins> } <del> return az.vmSet.GetInstanceIDByNodeName(string(name)) </del> <ins> return az.vmSet.GetInstanceIDByNodeName(nodeName) </ins> } // InstanceTypeByProviderID returns the cloudprovider instance type of the node with the specified unique providerID

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When useInstanceMetadata is set to true for vmss nodes, their providerID is wrong. What you expected to happen: Their providerID should be in format azure:///subscriptions/<subscription-id/resourceGroups/<rg-name/providers/Microsoft.Compute/virtualMachineScaleSets/<scaleset-name/virtualMachines/<instance-id How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
adding priority important-soon so that this issue can stay in 1.10. Please update if incorrect. /priority important-soon


Question: import ( "errors" "fmt" <ins> "k8s.io/kubernetes/cmd/kubeadm/app/constants" </ins> "path" "strings" "testing"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was using kubeadm to generate cluster ca: The CA generation failed because of 502 bad gateway returned and the response body isn't empty when it's trying to fetch stable version from internet. In this case, the function returns error instead of falling back to local client version: is how kubeadm fetches stable version and is how the error is handled for . Should kubeadm always fall back to client version when there is any internet issue? Otherwise, this sounds defeat the purpose of fallback. /sig cluster-lifecycle What you expected to happen: kubeadm falls back to local client version when there is any internet issue. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.13.7 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
I'd like to take this issue. It would be better if you could give some directions. /assign
recreate the issue in kubernetes/kubeadm: /close
Closing this issue. <detailsIn response to : recreate the issue in kubernetes/kubeadm: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the fix will be removing the when failed to and add some unit test if I don't missing anything
Got it. Thanks.


Question: # Uses the config file specified in $KUBE_CONFIG_FILE, or defaults to config-default.sh <del> KUBE_PROMPT_FOR_UPDATE=y KUBE_SKIP_UPDATE=${KUBE_SKIP_UPDATE-"n"} </del> <ins> KUBE_PROMPT_FOR_UPDATE=${KUBE_PROMPT_FOR_UPDATE:-"n"} </ins> KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. source "${KUBE_ROOT}/cluster/gke/${KUBE_CONFIG_FILE:-config-default.sh}" source "${KUBE_ROOT}/cluster/common.sh"

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.


Question: addProfilingFlags(flags) <del> kubeConfigFlags := genericclioptions.NewConfigFlags(true) </del> <ins> kubeConfigFlags := genericclioptions.NewConfigFlags(true).WithDeprecatedPasswordFlag() </ins> kubeConfigFlags.AddFlags(flags) matchVersionKubeConfigFlags := cmdutil.NewMatchVersionFlags(kubeConfigFlags) matchVersionKubeConfigFlags.AddFlags(cmds.PersistentFlags())

have worked since 1.0 as a way to pass basic auth credentials to the API server. In 1.11, these flags were unintentionally removed in Environment: Kubernetes version (use ): kubectl 1.11+ /sig cli /sig auth
/assign


Question: "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/rest:go_default_library", "//staging/src/k8s.io/component-base/featuregate/testing:go_default_library", <ins> "//test/integration/etcd:go_default_library", </ins> "//test/integration/framework:go_default_library", "//vendor/sigs.k8s.io/yaml:go_default_library", ],

What happened: I've performed server-side apply on the subresource. e.g. The request body is the following for example: The CRD itself has status commonly defined as: in the resulting openAPIv3 validation the correct object annotations are set for listType & listMapKey (thanks to controler-gen v0.2.5) What you expected to happen: The patch is applied to the status subresource, and the is properly set. However, one of two things happens: if the status subfield conditions don't exist the request passes through and properly sets the field. But the isn't set if the specific status condition type exits the request fails with 409 Conflict. It conflicts with other managers, e.g. since the is empty. if the exist beforehand for the status subresource, they are cleared after server-side apply on the subresource and become empty. Anything else we need to know?: This is similar to Also somewhat related to Environment: Kubernetes version (use ):
/wg apply
/sig api-machinery
thanks for the detailed post, you're absolutely right that this is related to the two issues that you mentioned. I don't see how we would be able to fix this in 1.18 or any version before that, we're planning on fixing it in 1.19. We should probably update the documentation to mention that this doesn't work for sub-resources, and maybe even disable it altogether for sub-resources.
I'd rather have it working...buggily(that is via ) than having it disabled altogether.
We trying to find out what this doesn’t work and see if we can back port the fix. We’ll keep you posted, thanks!
I can't reproduce this bug right now, would you mind sharing the CRD and exact reproduction steps?
Sure. The server-side application I've investigated in This is the CRD definition: my experimentations are in on line I've changed whether I've used force or not. The raw HTTP req/resp should be written to the output due to
Let me know if there's anything more I can do to help.
The strategy overriding the metadata in status updates of CRD and CR objects seems like it would definitely cause this. were you able to follow the steps at


Question: function detect-minions () { KUBE_MINION_IP_ADDRESSES=() for (( i=0; i<${#MINION_NAMES[@]}; i++)); do <ins> # gcutil will print the "external-ip" column header even if no instances are found </ins> local minion_ip=$(gcutil listinstances --format=csv --sort=external-ip  --columns=external-ip --zone ${ZONE} --filter="name eq ${MINION_NAMES[$i]}"  <del> | tail -n 1) echo "Found ${MINION_NAMES[$i]} at ${minion_ip}" KUBE_MINION_IP_ADDRESSES+=("${minion_ip}") </del> <ins> | tail --lines=+2 | tail -n 1) if [ -z "$minion_ip" ] ; then echo "Did not find ${MINION_NAMES[$i]}" 1>&2 else echo "Found ${MINION_NAMES[$i]} at ${minion_ip}" KUBE_MINION_IP_ADDRESSES+=("${minion_ip}") fi </ins> done if [ -z "$KUBE_MINION_IP_ADDRESSES" ]; then <del> echo "Could not detect Kubernetes minion nodes.  Make sure you've launched a cluster with 'kube-up.sh'" </del> <ins> echo "Could not detect Kubernetes minion nodes.  Make sure you've launched a cluster with 'kube-up.sh'" 1>&2 </ins> exit 1 fi }

I'm following the documentation on this doc: Everything works until this command: I get this error consistently from different VMs (a Debian and a CoreOS). Maybe I have to edit /etc/hosts to add the "external-ip" with the public IP of some server? Docs don't seem clear. Thanks, Riccardo
Its failing to look up your master correctly. You can try setting KUBERNETES_MASTER=http://<your IP hearBrendan On Sep 21, 2014 5:45 AM, "Riccardo Carlesso" wrote:


Question: framework.ExpectNoError(err, "Failed to patch ReplicationControllerScale") ginkgo.By("waiting for RC to be modified") eventFound = false <del> ctx, cancel = context.WithTimeout(context.Background(), 30*time.Second) </del> <ins> ctx, cancel = context.WithTimeout(context.Background(), 60*time.Second) </ins> defer cancel() _, err = watchUntilWithoutRetry(ctx, retryWatcher, func(watchEvent watch.Event) (bool, error) { if watchEvent.Type != watch.Modified {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: - Triage link for more jobs that are affected: Which test(s) are failing: Since when has it been failing: Since October 23 Testgrid link: Reason for failure: Anything else we need to know: Example spyglass link: /sig apps /cc /priority important-soon
/cc missed this in the main body
/assign /triage accepted
/remove-priority important-soon /priority critical-urgent top flake for pull-kubernetes-conformance-kind-ipv6-parallel, 2nd top flake for per /area conformance this is a conformance test /remove-kind failing-test /kind flake this is not failing continuously, but is instead flaking
/assign
I increased the timeout while waiting for available Replicas.
Test grid green since Merge 5 Nov 2020 /cc


Question: Ports: []api.ServicePort{{Port: 1234}}, }, } <del> err := handler.Admit(admission.NewAttributesRecord(newService, oldService, api.Kind("Service").WithVersion("version"), newService.Namespace, newService.Name, api.Resource("services").WithVersion("version"), "", admission.Update, nil)) </del> <ins> err := handler.Admit(admission.NewAttributesRecord(newService, existingService, api.Kind("Service").WithVersion("version"), newService.Namespace, newService.Name, api.Resource("services").WithVersion("version"), "", admission.Update, nil)) </ins> if err != nil { t.Errorf("Unexpected error: %v", err) }

With the migration to podutils and adding some presubmits (from periodic jobs) we are seeing some failures in presubmits. The error seems to have issues finding the right node to create.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /sig testing
any leads on solving this one?
see notes on slack -
/assign
this looks good now!


Question: __EOF__ # taint/untaint <del> # Pre-condition: node has no taints kube::test::get_object_assert "nodes 127.0.0.1" "{{.spec.taints}}" '<no value>' </del> <ins> # Pre-condition: node doesn't have dedicated=foo:PreferNoSchedule taint kube::test::get_object_assert "nodes 127.0.0.1" '{{range .spec.taints}}{{if eq .key "dedicated"}}{{.key}}={{.value}}:{{.effect}}{{end}}{{end}}' "" # expect no output </ins> # taint can add a taint kubectl taint node 127.0.0.1 dedicated=foo:PreferNoSchedule <del> kube::test::get_object_assert "nodes 127.0.0.1" '{{range .spec.taints}}{{.effect}}{{end}}' 'PreferNoSchedule' </del> <ins> kube::test::get_object_assert "nodes 127.0.0.1" '{{range .spec.taints}}{{if eq .key "dedicated"}}{{.key}}={{.value}}:{{.effect}}{{end}}{{end}}' "dedicated=foo:PreferNoSchedule" </ins> # taint can remove a taint kubectl taint node 127.0.0.1 dedicated- <del> # Post-condition: node has no taints kube::test::get_object_assert "nodes 127.0.0.1" "{{.spec.taints}}" '<no value>' </del> <ins> # Post-condition: node doesn't have dedicated=foo:PreferNoSchedule taint kube::test::get_object_assert "nodes 127.0.0.1" '{{range .spec.taints}}{{if eq .key "dedicated"}}{{.key}}={{.value}}:{{.effect}}{{end}}{{end}}' "" # expect no output </ins> ### kubectl cordon update with --dry-run does not mark node unschedulable # Pre-condition: node is schedulable

In autoscaling tests, we verify that when a node suddenly disappears from the cluster, Cluster Autoscaler will add another one to replace it. With TaintNodesByCondition (enabled by default in ), this is now failing: https://k8s- Every time a node disappears, a subset of pods become NodeLost and aren't rescheduled on a new node. /cc /sig node
I can work on this.
just got merged. Let's check result of autoscaling testings tomorrow: https://k8s-
latest autoscaling testing is green. I think we can close this issue.
Thanks for fixing this!


Question: # The business logic for whether a given object should be created # was already enforced by salt, and /etc/kubernetes/addons is the # managed result is of that. Start everything below that directory. <del> echo "== Kubernetes addon manager started at $(date -Is) ==" </del> KUBECTL=/usr/local/bin/kubectl <ins> function create-object() { obj=$1 for tries in {1..5}; do if ${KUBECTL} --server="127.0.0.1:8080" create --validate=true -f ${obj}; then return fi echo "++ ${obj} failed, attempt ${try} (sleeping 5) ++" sleep 5 done } echo "== Kubernetes addon manager started at $(date -Is) ==" </ins> for obj in $(find /etc/kubernetes/addons -name *.yaml); do <del> ${KUBECTL} --server="127.0.0.1:8080" create -f ${obj} & </del> <ins> create-object ${obj} & </ins> echo "++ addon ${obj} started in pid $! ++" done noerrors="true"

I thought I had worked this out during development, but from an IRC report from off a 0.10 GKE cluster, SkyDNS isn't starting. A bit of log trawling: Maybe I closed too early?
gist with additional info:
(FWIW, this came up the next time, and we test this setup all the time using "Services should provide DNS for the cluster" ) but even if this is a low grade flake, I take a bit of a personal affront.)
The cluster where SkyDNS didn't come up was created via the console at It was a cluster size of 2 on us-central1-b.


Question: return goerrors.New("error getting CSI client") } <del> var updateErrs []error </del> <ins> var lastErr error </ins> err := wait.ExponentialBackoff(updateBackoff, func() (bool, error) { <del> if err := nim.tryInitializeCSINodeWithAnnotation(csiKubeClient); err != nil { updateErrs = append(updateErrs, err) </del> <ins> if lastErr = nim.tryInitializeCSINodeWithAnnotation(csiKubeClient); lastErr != nil { klog.V(2).Infof("Failed to publish CSINode: %v", lastErr) </ins> return false, nil } return true, nil }) if err != nil { <del> return fmt.Errorf("error updating CSINode annotation: %v; caused by: %v", err, utilerrors.NewAggregate(updateErrs)) </del> <ins> return fmt.Errorf("error updating CSINode annotation: %v; caused by: %v", err, lastErr) </ins> } return nil

What happened: Upgraded from 1.16.6 to 1.17.2 and will no longer start on control-plane hosts. It fails with: What you expected to happen: to start, static pods to launch, to connect to in the static pod How to reproduce it (as minimally and precisely as possible): Have the , , , etc in the static pod manifest folder. Try to start the with the container images not preloaded on the host. Just in case it is needed: is used as the backend is run using Anything else we need to know?: This looks to be two things interacting poorly: 1- It takes a long time for static pods to be created for the first time when the can't talk to yet. 2- The new CSI migration code force terminates causing these log lines to appear in : It looks like introduced this new fatal error. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: Bare-metal OS (e.g: ): Fedora 31..1 (CoreOS preview)" Kernel (e.g. ): 5.3.12-300.fc31.x86_64 Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
Actually, probably: /sig node
The referenced PR was in 1.14, so that interaction is not new. What is likely new is the size increase of hyperkube in 1.17 (xref ). When not pre-pulled, this can increase the time it takes to start hyperkube for the first time. Does the image pull make progress as the kubelet restarts/retries?
It might make sense for the initializeCSINode() method to: first wait to successfully talk to the kube apiserver at all, perhaps indefinitely after successfully talking to the kube apiserver, then try to register the CSINode object with backoff, and fail if it cannot do that successfully
/sig storage /remove-sig cluster-lifecycle
It doesn't seem to. Once the connection to containerd is closed, it cancels the context. Otherwise I would expect this to succeed eventually if the pull happened in the background. That sounds reasonable to me assuming it isn't blocking the static pod creation code. Do you happen to know why it takes a very long time for the static pod creation code to be called?

I would expect the pods to be created very soon after the kubelet is started.
Pulling the image and starting the processes for those pods happens immediately (from the containerd logs, you can see the image pull happening... it just doesn't complete in time before the independent kubelet logic to register the CSI node info times out).
Ah, yes. I just upgraded a couple of days ago so I didn't check whether I still saw the 2 minute delay in creating the "pod sandboxes" (this is before the pull even happens). It now is only a 23 second delay: containerd: kubelet:
ack, I like the idea Will work on a fix today and cherry-pick it as necessary


Question: return err } <ins> // Drop EndpointSlices that have been marked for deletion to prevent the controller from getting stuck. endpointSlices = dropEndpointSlicesPendingDeletion(endpointSlices) </ins> if c.endpointSliceTracker.StaleSlices(service, endpointSlices) { return endpointsliceutil.NewStaleInformerCache("EndpointSlice informer cache is out of date") }

As discussed in , Endpoint(Slice) controllers can get stuck if an Endpoints or EndpointSlice resource is stuck in a terminating state. I don't think there's much we can do for Endpoints here since the same name is reused, but EndpointSlice controllers can ignore EndpointSlices in a terminating state. See No response All supported versions Any Any Any Any Any
/sig network
/triage accepted
deleting a service with a finalizer doesn't affect the endpointslice, it doesn't change its status to termination or unready Repro: finalizer to the service and delete it, it keeps waiting until the finalizer is removed with the deletion timestamp set. the endpoint slice associated Do you mean that when the endpoint slice controller detects that the Service is being deleted should switch the endpoint slice status to Terminating? cc:
I don't think we need to do anything special if the Service is waiting to be deleted. There are many legitimate reasons for this to happen, including waiting for infrastructure to be removed. My main concern with a finalizer was that one on an EndpointSlice could cause the controller to get stuck on a Service. We don't have the same risk with a finalizer on a Service - I don't think that would cause the controller to get stuck in any way.
i was unable to repro the issue on my machine: I was monitoring the logs of , and the endpointslice controller never complained about , when I deleted the service. Though, I was able to get the controller to log this error under a different scenario. If we don't explicitly remove the finalizer from the endpointslice, and instead re-create the pod and service like: From what I can understand, because the endpointslice with the finalizer is owned by a service with a different UID than the new endpointslice, it is marked as expected for deletion . The endpointslice controllers calls , which returns true for the endpointslice with the finalizer, because of the above reason. I might be wrong anywhere here, so please let me know if that is the case. As for further action, I'm not sure what can be done here? We could check the endpointslice for finalizers and then skip them . Keen to hear your thoughts about this.


Question: if obj.Spec.Unschedulable { status = append(status, "SchedulingDisabled") } <ins> role := findNodeRole(obj) if role != "" { status = append(status, role) } </ins> row.Cells = append(row.Cells, obj.Name, strings.Join(status, ","), translateTimestamp(obj.CreationTimestamp), obj.Status.NodeInfo.KubeletVersion) if options.Wide {

used to show masters, but as of it no longer does. This is not in accordance with our deprecation policy I believe. cc
That commit/PR is not relevant as there are no supported users/deployment methods that are using label. kubeadm v1.4 or v1.5 was the last. What you're really looking for is who/what removed printing things based on the label...
No blame on you The regression is that we have stopped printing masters in . There may be more to it that a simple revert of , but we should probably start there.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This has been fixed in recent versions


Question: } }) <del> It("should list pulled images [Conformance]", func() { </del> <ins> It("should list pulled images", func() { </ins> image, _ := NewConformanceImage("docker", "") tags, err := image.List() Expect(err).ShouldNot(HaveOccurred())

As of now node e2e requires all the tests to pass. There are tests like the container conformance tests which have been flaky in the past - , , We need ability to mark certain tests as flaky and not run them as part of merge queue.
Ginkgo flags makes it possible to skip tests. We can use a prefix to not run flaky tests in our existing jenkins CIs.
Closing via


Question: "//staging/src/k8s.io/client-go/informers:go_default_library", "//staging/src/k8s.io/client-go/kubernetes/fake:go_default_library", "//staging/src/k8s.io/cloud-provider:go_default_library", <ins> "//vendor/github.com/vmware/govmomi:go_default_library", </ins> "//vendor/github.com/vmware/govmomi/lookup/simulator:go_default_library", "//vendor/github.com/vmware/govmomi/property:go_default_library", "//vendor/github.com/vmware/govmomi/simulator:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Provsioning a PVC using a storage class with vSphere as the cloud provider failed with error "runtime error: index out of range". What you expected to happen: The PVC should have been created. How to reproduce it (as minimally and precisely as possible): Create a PVC using a storage class with vSphere as the cloud provider and a VMware vCenter Server where the default language is not English. Anything else we need to know?: K8s in conjunction with vSphere PVC relies on English error messages from the VMware API in order to determine the canonical path for a newly created volume (VMDK file). The faulting code (v1.11.3, still the same on master) can be found here (starting at line 312): The line "canonicalVolumePath = match[1]" (340) in the above code is the problem because "File (.*?) was not found" (English) never matches for a different language (e.g. German: "Datei [DATASTORE] kubevols/kube- wurde nicht gefunden") and a “runtime error: index out of range” oocurs in the "kube-controller-manager". This causes the already created volume (VMDK file) not to be reported back to the K8s system which in turn will try to recreate it repeatedly in a loop (fails with "Cannot complete the operation because the file or folder [DATASTORE] /kubevols/kubernetes-dynamic-pvc-[UUID] already exists"). Find below the corresponding log excerpt (kube-controller-manager): Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): n/a Kernel (e.g. ): n/a Install tools: n/a Others: n/a <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign


Question: klog.Errorf("Unable to lookup pv object for: %q, err: %v", *pvName, err) continue } <ins> var plugin volume.AttachableVolumePlugin </ins> volumeSpec := volume.NewSpecFromPersistentVolume(pv, false) <del> plugin, err := adc.volumePluginMgr.FindAttachablePluginBySpec(volumeSpec) if err != nil || plugin == nil { // Currently VA objects are created for CSI volumes only. nil plugin is unexpected, generate a warning klog.Warningf( "Skipping processing the volume %q on nodeName: %q, no attacher interface found. err=%v", *pvName, nodeName, err) continue </del> <ins> // Consult csiMigratedPluginManager first before querying the plugins registered during runtime in volumePluginMgr. // In-tree plugins that provisioned PVs will not be registered anymore after migration to CSI, once the respective // feature gate is enabled. if inTreePluginName, err := adc.csiMigratedPluginManager.GetInTreePluginNameFromSpec(pv, nil); err == nil { if adc.csiMigratedPluginManager.IsMigrationEnabledForPlugin(inTreePluginName) { // PV is migrated and should be handled by the CSI plugin instead of the in-tree one plugin, _ = adc.volumePluginMgr.FindAttachablePluginByName(csi.CSIPluginName) // podNamespace is not needed here for Azurefile as the volumeName generated will be the same with or without podNamespace volumeSpec, err = csimigration.TranslateInTreeSpecToCSI(volumeSpec, "" /* podNamespace */, adc.intreeToCSITranslator) if err != nil { klog.Errorf( "Failed to translate intree volumeSpec to CSI volumeSpec for volume:%q, va.Name:%q, nodeName:%q: %s. Error: %v", *pvName, va.Name, nodeName, inTreePluginName, err) continue } } </ins> } <del> pluginName := plugin.GetPluginName() if adc.csiMigratedPluginManager.IsMigrationEnabledForPlugin(pluginName) { plugin, _ = adc.volumePluginMgr.FindAttachablePluginByName(csi.CSIPluginName) // podNamespace is not needed here for Azurefile as the volumeName generated will be the same with or without podNamespace volumeSpec, err = csimigration.TranslateInTreeSpecToCSI(volumeSpec, "" /* podNamespace */, adc.intreeToCSITranslator) if err != nil { klog.Errorf( "Failed to translate intree volumeSpec to CSI volumeSpec for volume:%q, va.Name:%q, nodeName:%q: %v", </del> <ins> if plugin == nil { plugin, err = adc.volumePluginMgr.FindAttachablePluginBySpec(volumeSpec) if err != nil || plugin == nil { // Currently VA objects are created for CSI volumes only. nil plugin is unexpected, generate a warning klog.Warningf( "Skipping processing the volume %q on nodeName: %q, no attacher interface found. err=%v", </ins> *pvName, <del> va.Name, </del> nodeName, <del> pluginName, </del> err) continue } } volumeName, err := volumeutil.GetUniqueVolumeNameFromSpec(plugin, volumeSpec) if err != nil { klog.Errorf(

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Context: When a cluster has dangling s caused by restarts of kube-controller-manager (like described in ), the attach/detach controller tries to clean them up on startup (ref ). For this, it iterates over all s and finds the responsible volume plugin for retrieving the unique volume name. Problem: However, if the volume was provisioned by an in-tree plugin, then later migrated to CSI and now the / feature gate is set, the volume is skipped during , so the will stay in the system and the volume will not be detached from the undesired node and potentially block a new attach operation to a different node. Analysis: The / feature gate causes the respective in-tree volume plugins not to be registered: The spec only contains the in-tree fields (e.g or ), but not the fields: This causes the call in to find no responsible volume plugin for the , which will skip the cleanup: That's why we see a lot of logs like this on startup of kube-controller-manager and dangling s on our clusters: The attach/detach controller should cleanup dangling s for migrated in-tree plugins with unregistered in-tree plugins as well. ref , addresses a similar issue. Though, the fix doesn't come into effect, if the respective / feature gate is enabled. Kubernetes version (use ): / Cloud provider or hardware configuration: AWS, GCP
/sig storage cc
Hey, thanks for reporting this bug!! I will try to come up with a fix. BTW, wondering what's the symptom you observed except for the log. Is there any volume operation(attaching/detaching/mounting/unmounting/creation/deletion) not working? This, to my understanding, will only make dangling attachment not working properly but I guess normal operation should work, is it?
The consequence of the leftover s in our case is that volumes are not detached on infrastructure provider side when moving pods with volumes between nodes, e.g.

because of rolling updates/eviction/etc.. This in turn causes the new s on the new nodes to fail with logs similar to this one: (we observe the same for GCP) I can also work on a fix tomorrow, if you want. I think, we should be able to reuse the fix from and additionally determine the from or similar, instead of only relying on .
That would be nice! Really appreciate it. Yep, sounds like a good idea. If you could also help to verify the fix in your environment that would be awesome
Ok, great. Will try to come up with a fix. /assign Definitely! I'll try my best and let you know if I need some help :)
Yep, we observed this after experiencing the bug. We ended up manually creating a cleanup script to check EC2 <-K8s VA if it was actually attached or not, otherwise delete VA. My understanding is with fixed we will no longer have this issue, but it would be nice to have the garbage collection logic work for migrated/non-migrated volumes regardless
/triage accepted


Question: creationTimestamp: null labels: kubernetes.io/bootstrapping: rbac-defaults <ins> name: system:aws-cloud-provider rules: - apiGroups: - "" resources: - nodes verbs: - get - patch - apiGroups: - "" resources: - events verbs: - create - patch - update - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" creationTimestamp: null labels: kubernetes.io/bootstrapping: rbac-defaults </ins> name: system:basic-user rules: - apiGroups:

Recently introduced fix of tainting the node if EBS volume on a node is stuck for too long in "attaching" state does not work on RBAC enabled clusters because of missing service account permissions. /sig storage /sig aws