Question: if v1helper.GetPersistentVolumeClass(volume) != requestedClass { continue } <ins> if !nodeAffinityValid { continue } </ins> if node != nil { // Scheduler path

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: Volume got bound to PV on node "kubernetes-minion-group-82c8" When the pod went back through scheduling, the predicate returned success for other nodes even though we should have updated the PV cache to point to the PVC: I think in , instead of continuing if the node affinity doesn't match, it should fail if the PV is prebound.
This causes two issues: On the second pass, the scheduler may choose a different node. Then in Assume/Bind volumes, we'll also prebind a second set of PVs. If the scheduler chose a different node on the second pass, and the PVCs became fully bound in between the predicates and assume volumes, then assume volumes will return success and the pod will get scheduled to the wrong node.


Question: } } <del> // TestApplyCRDNonStructuralSchema tests that when a CRD has a non-structural schema in its validation field, // it will be used to construct the CR schema used by apply, but any non-structural parts of the schema will be treated as // nested maps (same as a CRD without a schema) func TestApplyCRDNonStructuralSchema(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ServerSideApply, true)() server, err := apiservertesting.StartTestServer(t, apiservertesting.NewDefaultTestServerOptions(), nil, framework.SharedEtcd()) if err != nil { t.Fatal(err) } defer server.TearDownFn() config := server.ClientConfig apiExtensionClient, err := clientset.NewForConfig(config) if err != nil { t.Fatal(err) } dynamicClient, err := dynamic.NewForConfig(config) if err != nil { t.Fatal(err) } noxuDefinition := fixtures.NewNoxuCustomResourceDefinition(apiextensionsv1beta1.ClusterScoped) var c apiextensionsv1beta1.CustomResourceValidation err = json.Unmarshal([]byte(`{ "openAPIV3Schema": { "type": "object", "properties": { "spec": { "anyOf": [ { "type": "object", "properties": { "cronSpec": { "type": "string", "pattern": "^(d+|*)(/d+)?(s+(d+|*)(/d+)?){4}$" } } }, { "type": "string" } ] } } } }`), &c) if err != nil { t.Fatal(err) } noxuDefinition.Spec.Validation = &c noxuDefinition, err = fixtures.CreateNewCustomResourceDefinition(noxuDefinition, apiExtensionClient, dynamicClient) if err != nil { t.Fatal(err) } kind := noxuDefinition.Spec.Names.Kind apiVersion := noxuDefinition.Spec.Group + "/" + noxuDefinition.Spec.Version name := "mytest" rest := apiExtensionClient.Discovery().RESTClient() yamlBody := []byte(fmt.Sprintf(` apiVersion: %s kind: %s metadata: name: %s finalizers: - test-finalizer spec: cronSpec: "* * * * */5" replicas: 1`, apiVersion, kind, name)) result, err := rest.Patch(types.ApplyPatchType). AbsPath("/apis", noxuDefinition.Spec.Group, noxuDefinition.Spec.Version, noxuDefinition.Spec.Names.Plural). Name(name). Param("fieldManager", "apply_test"). Body(yamlBody). DoRaw(context.TODO()) if err != nil { t.Fatalf("failed to create custom resource with apply: %v:n%v", err, string(result)) } verifyNumFinalizers(t, result, 1) verifyFinalizersIncludes(t, result, "test-finalizer") verifyReplicas(t, result, 1.0) // Patch object to add another finalizer to the finalizers list result, err = rest.Patch(types.MergePatchType). AbsPath("/apis", noxuDefinition.Spec.Group, noxuDefinition.Spec.Version, noxuDefinition.Spec.Names.Plural). Name(name). Body([]byte(`{"metadata":{"finalizers":["test-finalizer","another-one"]}}`)). DoRaw(context.TODO()) if err != nil { t.Fatalf("failed to add finalizer with merge patch: %v:n%v", err, string(result)) } verifyNumFinalizers(t, result, 2) verifyFinalizersIncludes(t, result, "test-finalizer") verifyFinalizersIncludes(t, result, "another-one") // Re-apply the same config, should work fine, since finalizers should have the list-type extension 'set'. result, err = rest.Patch(types.ApplyPatchType). AbsPath("/apis", noxuDefinition.Spec.Group, noxuDefinition.Spec.Version, noxuDefinition.Spec.Names.Plural). Name(name). Param("fieldManager", "apply_test"). SetHeader("Accept", "application/json"). Body(yamlBody). DoRaw(context.TODO()) if err != nil { t.Fatalf("failed to apply same config after adding a finalizer: %v:n%v", err, string(result)) } verifyNumFinalizers(t, result, 2) verifyFinalizersIncludes(t, result, "test-finalizer") verifyFinalizersIncludes(t, result, "another-one") // Patch object to change the number of replicas result, err = rest.Patch(types.MergePatchType). AbsPath("/apis", noxuDefinition.Spec.Group, noxuDefinition.Spec.Version, noxuDefinition.Spec.Names.Plural). Name(name). Body([]byte(`{"spec":{"replicas": 5}}`)). DoRaw(context.TODO()) if err != nil { t.Fatalf("failed to update number of replicas with merge patch: %v:n%v", err, string(result)) } verifyReplicas(t, result, 5.0) // Re-apply, we should get conflicts now, since the number of replicas was changed. result, err = rest.Patch(types.ApplyPatchType). AbsPath("/apis", noxuDefinition.Spec.Group, noxuDefinition.Spec.Version, noxuDefinition.Spec.Names.Plural). Name(name). Param("fieldManager", "apply_test"). Body(yamlBody). DoRaw(context.TODO()) if err == nil { t.Fatalf("Expecting to get conflicts when applying object after updating replicas, got no error: %s", result) } status, ok := err.(*apierrors.StatusError) if !ok { t.Fatalf("Expecting to get conflicts as API error") } if len(status.Status().Details.Causes) != 1 { t.Fatalf("Expecting to get one conflict when applying object after updating replicas, got: %v", status.Status().Details.Causes) } // Re-apply with force, should work fine. result, err = rest.Patch(types.ApplyPatchType). AbsPath("/apis", noxuDefinition.Spec.Group, noxuDefinition.Spec.Version, noxuDefinition.Spec.Names.Plural). Name(name). Param("force", "true"). Param("fieldManager", "apply_test"). Body(yamlBody). DoRaw(context.TODO()) if err != nil { t.Fatalf("failed to apply object with force after updating replicas: %v:n%v", err, string(result)) } verifyReplicas(t, result, 1.0) } </del> // verifyNumFinalizers checks that len(.metadata.finalizers) == n func verifyNumFinalizers(t *testing.T, b []byte, n int) { obj := unstructured.Unstructured{}

With in a CRD we get: This is because gnostic's OpenAPI proto types have no nullable support. Luckily CRDs keep working because in the upper error case we don't stop serving, just set the SSA schema to nil.
Thanks for reporting!
I think it should be "This is because gnostic's OpenAPI v2 proto types have no nullable support.". The problem is that we're not supposed to pass-in v3.
Source:
OK, We'll write a test and fix this, thanks Stefan!
/reopen This still happens in slightly different shape: This comes from :
That's a fairly different bug but yes, thanks!
Is this related to the stating that use of "anyOf" wouldn't be allowed after 1.16? We're seeing issue on OpenFaaS after going to 1.17.5 and 1.18. Usually upgrading K8s versions hasn't caused breaking changes and we're trying to figure out where to start with this one.
No. That post was about structural schemas. v1 CRDs are limited to structural schemas. v1beta1 CRDs can continue to create non-structural schemas until v1beta1 is no longer served (in 1.22) Also, is not completely disallowed, specific restrictions inside an clause (like ) are what are not allowed ("logical constraints using oneOf, allOf, anyOf, not are allowed").
extracted the newly found issue to to track separately with isolated reproducer /close
Closing this issue. <detailsIn response to : extracted the newly found issue to to track separately Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return 1 } <ins> kube::util::download_file() { local -r url=$1 local -r destination_file=$2 rm  ${destination_file} 2&> /dev/null || true for i in $(seq 5) do if ! curl -fsSL --retry 3 --keepalive-time 2 ${url} -o ${destination_file}; then echo "Downloading ${url} failed. $((5-i)) retries left." sleep 1 else echo "Downloading ${url} succeed" return 0 fi done return 1 } </ins> # ex: ts=2 sw=2 et filetype=sh

Sometimes e2e tests fails during cluster installation because curl fails with: For example see here: https://k8s- Downloading etcd failed. I've run curl to get etcd from github 4400 times. It failed 5 times. So it doesn't happen often.


Question: {"GET", "/api", "", code200}, {"GET", "/healthz", "", code200}, {"GET", "/version", "", code200}, <ins> {"GET", "/invalidURL", "", code404}, </ins> } return requests }

Authenticated requests to master with an invalid URL path should return a non-status 200 response code. Preferably a 404 (not found). Sample output: { "paths": [ "/api", "/api/v1beta1", "/api/v1beta2", "/api/v1beta3", "/healthz", "/healthz/ping", "/logs/", "/metrics", "/static/", "/swagger-ui/", "/swaggerapi/", "/validate", "/version" ] }


Question: pathOptions.EnvVar = "" buf := bytes.NewBuffer([]byte{}) cmd := NewCmdConfigUnset(buf, pathOptions) <del> cmd.SetArgs(test.args) if err := cmd.Execute(); err != nil { t.Fatalf("unexpected error executing command: %v,kubectl unset args: %v", err, test.args) </del> <ins> opts := &unsetOptions{configAccess: pathOptions} err = opts.complete(cmd, test.args) if err == nil { err = opts.run(buf) </ins> } config, err := clientcmd.LoadFromFile(fakeKubeFile.Name()) if err != nil { t.Fatalf("unexpected error loading kubeconfig file: %v", err) } <ins> if err != nil && err.Error() != test.expectedErr { t.Fatalf("expected error:n %vnbut got error:n%v", test.expectedErr, err) } </ins> if len(test.expected) != 0 { if buf.String() != test.expected { t.Errorf("Failed in :%qn expected %vn but got %v", test.description, test.expected, buf.String())

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): 1.5.3 Environment: Cloud provider or hardware configuration: minikube 0.17.1 How to reproduce it (as minimally and precisely as possible): config Note that there's only one context. the property of a non-existent context config Note that a context named foo was created just because I tried to unset a property. What you expected to happen: should give an informative message saying that context doesn't exist. Anything else we need to know: This is probably low-hanging fruit.
/sig cli
/assign i will take a look for this
GitHub didn't allow me to assign the following users: a, look, for, this, i, will, take. Note that only can be assigned. <detailsIn response to : /assign i will take a look for this Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i can reproduce this, this is a bug, have open PR fix this. thanks
/kind bug


Question: } // assignPod assigns the given pod to the given machine. <del> func (r *BindingREST) assignPod(ctx context.Context, podID string, machine string, annotations map[string]string, dryRun bool) (err error) { if _, err = r.setPodHostAndAnnotations(ctx, podID, "", machine, annotations, dryRun); err != nil { </del> <ins> func (r *BindingREST) assignPod(ctx context.Context, podUID types.UID, podResourceVersion, podID string, machine string, annotations map[string]string, dryRun bool) (err error) { if _, err = r.setPodHostAndAnnotations(ctx, podUID, podResourceVersion, podID, "", machine, annotations, dryRun); err != nil { </ins> err = storeerr.InterpretGetError(err, api.Resource("pods"), podID) err = storeerr.InterpretUpdateError(err, api.Resource("pods"), podID) if _, ok := err.(*errors.StatusError); !ok {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The api bind a pod to a node. But the pod is specified by name (not uid), which is not a unique identifier for pod. So the caller of can't make sure the pod is exactly what he want. api can bind a pod with specified uid to a node. If the uid not match, return an error a pending pod by statefulset named busybox, make sure it's pending, the scheduler will not bind it(use non-existed node selector or other method). So we will get a pending pod busybox-0. following code the in the code, delete the pod busybox-0 manually during the sleep or other method. The statefulset busybox will create another pod busybox-0 with different uid , the api in the code above will bind an unexpected pod to node The logic in scheduler is the same. If the pod is recreate with the same name and different resource require or selector between scheduler's and , scheduler will do a wrong bind for the pod. I think the root cause is the api does not check the uid. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.0-alpha.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-20T09:01:41Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.9-aliyunedge.1", GitCommit:"", GitTreeState:"", BuildDate:"2021-03-08T09:56:01Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/sig scheduling for triage (the binding feature of (pod to node) is for scheduling).
For the sake of precedence, are there other APIs that are based on object UID?
uids are used as a precondition is used in many places. if the server started checking that a uid on the binding object (if specified) matched the uid of the pod being modified, that seems sufficient
is there any way for the caller of Bind to confirm that he is binding the right pod? or not uid, use resource version?
that would require: the server to check that bind UID (if specified) matches the pod UID (passing the binding uid from BindingREST#Create → assignPod → setPodHostAndAnnotations and verifying the uid match in setPodHostAndAnnotations) the binding API caller to set in the posted binding object (as illustrated in the description of this issue)
actually, kube-scheduler(the most common binding API caller) has set the in the posted binding object. but the server doesn't check the uid. in my program, i do those to work around(not elegant, but it works...): call the binding API with an uid in annotation(i found that BindingREST#Create → assignPod → setPodHostAndAnnotations will carry annotation). check the matchs , if not match, delete the pod and wait for an new pod ... from your point of view, is it necessary to ensure the pod is the caller want when someone call bind API? what's your advice for this issue
given the scheduler made decisions based on the current shape of the pod spec, I think it makes sense for it to work to make sure it is binding that instance of the pod I would advocate for supporting uid as a precondition in the binding API and starting to use it from the scheduler
i agree with you
/remove-sig api-machinery
I also agree with this and this cannot achieve only on sig/scheduling. Sorry, if it is not suitable, but I'd like to add sig/api-machinery label again.

/sig api-machinery
Either scheduling or node sigs own the pod binding API. Either of those would be a good place to coordinate adding a fix to honor uid as a precondition API machinery owns the generic API plumbing and API server behavior, not implementations of most specific APIs like the pod or pod binding APIs /remove-sig api-machinery
Given you already exercised Jodan's suggestion locally, would you mind raising an offical PR?
how about this PR


Question: type ContainerImage struct { // Names by which this image is known. // e.g. ["k8s.gcr.io/hyperkube:v1.0.7", "dockerhub.io/google_containers/hyperkube:v1.0.7"] <ins> // +optional </ins> Names []string `json:"names" protobuf:"bytes,1,rep,name=names"` // The size of the image in bytes. // +optional

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --is defined as For , both and are valid and they will be serialized to and in Json respectively. However in openapi-, is marked as required so is not allowed. This cause a mismatch when doing deserialization based on openapi spec. should not be marked as required because is a valid value. Kubernetes version (use ): v1.19 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node api-machinery
/triage accepted /assign
Hello, is there any update for this issue?
Possibly related to and This bug can prevent a controller we're working on from enumerating nodes in a cluster, as we hit deserialization errors if any node in the cluster exposes an image with . It seems like proposed to address this by using the image ID instead of its name. I imagine that making this field optional would require a KEP, as it's an API change; but perhaps this could be fixed by ensuring that the field is always initialized (i.e. to an empty list) to prevent deserialization errors? This would not require an API change and would make the implementation conform with the API spec.
Looking at the sources, it looks like the image list is constructed here: It's not clear to me how it's even possible for to be uninitialized? I'll try to figure out how we got into this state...
I was just able to trigger this with though I imagine this can be replicated in other environments (at least in ) with a similar workflow: To get into this state, I first built my app with and then imported it into the local k3d cluster with . Then, I made some changes to my application and built it again with the same tag, and imported into the cluster with the same command. It's not clear whether the instance represents the first version of the application image and the named instance refers to the second build on the same tag, or vice versa.
Sorry for the late reply. That's interesting. I agree with you it seems impossible for to be nil.

<- doesn't seem to be the case. Looking
I can reproduce it by changing to . Either I missed something basic or there is a bug in the (kubelet uses v1 types directly) serialization pipeline.
I think I understand what happened: If an image has neither nor , kubelet sets to . However, when the object gets persisted to etcd in protobuf, the data is stored as -- protobuff doesn't distinguish and : Although we ask JSON API clients to tolerate values for such fields, openapi-generated clients (e.g. the ) treat as an invalid value for a required field during deserialization. It's hard to tell what the right fix is: it's a data lost in kube-apiserver serialization/deserialization IMO, but it may be hard to fix, since was the rollback of an attempt to fix the field as optional is an option, but it needs to be approved by someone from SIG Node (and for other APIs that may have similar issue, it would depend on the implementation details of each API). On the plus side, there seems to be no server-side validation that requires to be not . kubelet is the only writer of the field, and the fact is sometimes the field is written as the JSON API client tolerate value (as suggested by ) is an option, but it has to be done in a client-by-client basis. For , I think we can maintain a patch (namely treating as ) in the Python client to fix this. I'm not sure what client you're using though Please note that this could also be fixed if the node API takes a different approach when an image has neither nor , e.g. using the image ID as the name (could be a breaking change to existing clients that tolerates ) assigning a dummy name like <none(which could be confusing if there are multiple such images / an image that is literally called <none)
I'm not familiar enough with the internals of k8s' etcd protobuf serialization to know whether this is feasible in this context, but in other applications I've used wrapper messages to avoid this ambiguity. For example: This lets the value-ness of the names field be encoded distinctly from the list itself.

Another instance of this issue discussed here
Kubernetes python client fix is tracked in contains some API change suggestions for SIG node to consider, which could benefit JSON API clients: /unassign /remove-sig api-machinery /remove-triage accepted let's see if someone from SIG node can pick this up
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks! But isn't that a breaking change if an old client talks to a new server? ah awesome! Probably we should just send a PR to SIG node to mark the field as optional /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : thanks! But isn't that a breaking change if an old client talks to a new server? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign actually let me open one real quick
As I said, I'm not really clear on where etcd protobuf clients are implemented or how public that API is. Generally, though, it could be done gracefully via a deprecation cycle. A new field is introduced and the prior field is considered deprecated; Writers write both the deprecated field and the new type field; The API server continues to read the deprecated field for one major version. Once a major version has passed, change the API server to read the new typed field and stop writing the deprecated version... But this is all way more complicated than marking the field as optional. If it's acceptable to make it optional, that sounds fine to me! Thanks for your help looking into this
that makes sense. Our proto definitions and clients are generated from the files, so supporting the deprecation logic in the clients does sound like more work
Thanks for the fix I am using .

it turns out that already has a fix on master ( -- not yet on ) and I've put up a PR to make it usable from kube-rs ()


Question: func loadWebhook(configFile string, groupVersion schema.GroupVersion, initialBackoff time.Duration, customDial utilnet.DialFunc) (*webhook.GenericWebhook, error) { w, err := webhook.NewGenericWebhook(audit.Scheme, audit.Codecs, configFile, []schema.GroupVersion{groupVersion}, initialBackoff, customDial) <ins> if err != nil { return nil, err } </ins> w.ShouldRetry = retryOnError <del> return w, err </del> <ins> return w, nil </ins> } type backend struct {

What happened: When parsing an audit webhook config file that errors (ca file does not exist), the apiserver will panic. What you expected to happen: Error gracefully with explanation to the error (ca authority does not exist). How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: we are assigning a null pointer when . We just need to add a standard before. Will follow up with a PR shortly. Environment: Kubernetes version (use ):kube-apiserver --version Kubernetes v1.19.0-alpha.0.597+ /kind bug /area apiserver
/sig api-machinery


Question: if !isInternal && wantLb { // Confirm desired public ip resource exists var pip *network.PublicIPAddress <del> domainNameLabel := getPublicIPDomainNameLabel(service) if pip, err = az.ensurePublicIPExists(service, desiredPipName, domainNameLabel, clusterName, shouldPIPExisted); err != nil { </del> <ins> domainNameLabel, found := getPublicIPDomainNameLabel(service) if pip, err = az.ensurePublicIPExists(service, desiredPipName, domainNameLabel, clusterName, shouldPIPExisted, found); err != nil { </ins> return nil, err } return pip, nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux


Question: }`, expectedCorefileData: `.:53 { errors <ins> logs </ins> health { lameduck 5s }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: does not retaining coredns configmap using v1.19.4 makes no difference What you expected to happen: should follow documentation and retain coredns configmap How to reproduce it (as minimally and precisely as possible): upgrading from v1.18.6 to v1.19.3 Anything else we need to know?: works like a charm Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: OS (e.g: ): CentOS 7 Kernel (e.g. ): 4.20.13- Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
seems to be a regression ?
/assign
Would you check you dns configmap for ? As , federation is removed from coredns. If you have federation enabled in your coredns configmap, the check may fail in
We don't have any parameter in our coredns configmap, or any kube-dns parameters in kubeadm config
Looking closely at the dry run messages above, this may not be a regression. The apparent reason it works in 1.18.12 is that support for CoreDNS 1.7.0 was not until 1.19. i.e. 1.18 sees an unsupported version of CoreDNS and therefore skips the corefile migration. Whereas in 1.19, CoreDNS 1.7.0 is supported, and the migration takes place. What does your complete Corefile look like before the migration?
If it's possible, can you also attach the entire dry run logs?
Sure.
Thanks.

I think the issue may be in the detection of the "default" configuration. That is, in the upgrade process, if it detects that you are using the "default" coredns corefile, it simply installs the new default over top of it, rather than migrating the existing one. I think since the default config stanza is in there in its original form, the whole Corefile is incorrectly being detected as a default. If i'm right, the fix could either go in the package (should at least eventually be fixed there), and/or kubeadm could just always migrate the existing config instead of deploying the default over top of the existing one.
I just tested against the package manually - and it correctly identifies the config as "non-default", so that's not it... And I also tested a migration on the Corefile, and the result included the extra stanzas as expected. So, the issue doesn't appear to lie in the migration package.
After testing with the code, I find the issue in , I fixed it in . A workaround for you is to create an empty kube-dns configmap. ———————————————————— edited new comments here I find still fix a problem , when kube-dns configmap is using federations.
Thanks, I'll try it
empty kube-dns configmap doesn't helped.
sorry for my mistake. According to my test, the create or update will trigger is no migrating. Even your corefile will meet some dep that need to be migrated. max_concurrent is new default. However, there is a typo in . Below is a workaround(not good one), add upstream to your corefile in kubernetes. ! It works for me.
if your current coredns is 1.6.7, upstream is deprecated but not removed. In new 1.7.0 coredns, upstream is removed.
thanks for the discussion and PRs this one is mergable for 1.21 as non critical: what do you think about this one and should we backport it as critical (also, to which k8s versions)?
/area kubeadm
It's not obvious to me how this relates to . The kubeadm dry run log appears to be replacing the existing config with the default. It should only be doing this if the existing config is semantically the same as a default Corefile, i.e.

if is true.
just explains that the migration is not triggered at all. if is not a fix for this issue, please remove this line from the PR description.
I think issue here is in/around , and how the result is interpreted... The code correctly assumes that if a Corefile is a default Corefile, then no migration is required. However later it incorrectly assumes the reciprocal: If migration is not required, then the corefile must be a default. Hence, it ends up replacing any Corefile that does not require migration with a default… even if it wasn't a default to begin with.
yes. The need be fixed as well, I will learn to fix it later if no one is assigned. I changed releaseNote to None.


Question: rcInstance.volumesNeedUpdateFromNodeStatus = nil // Act 2 - reconcile once <del> rcInstance.reconcileNew() </del> <ins> rcInstance.reconcile() </ins> // Assert 2 // MountDevice was attempted

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We have even encountered such case that pod continuously reports “Unable to attach or mount volumes for pod;...” when kubelet restarted. The reasons are listed in details as below: , as result of apiserver instantaneous unstablility, dswp get pvc‘s pv spec failed, then the pvc volume does not be to dsw. , syncStates() lists all the volume directories locally, and recontructs the volume from local volume, if the volume isn't in dsw, it will put the recontructed volume to asw, but now the volume.OuterVolumeSpecName isn't correct. chance, at this time, In , the fail to pvc volume is to dsw again normally in reconcile() -mountAttachVolumes(), it compares dsw and asw by volume's innerVolumename, it's all the same. but it could not sense the OuterVolumeSpecName dismatch. so it will not re-mount again to update asw cache , it checks pod's volume attach or mount success by comparing volume.OuterVolumeSpecName and , but this pv'c OuterVolumeSpecName isn't correct in asw. so pod continuously reports “Unable to attach or mount volumes for pod;...” reconcile() could sense the reconstruct volume's dismatch to the dsw, then re-mount again to make volume cache consistency Kubernetes version (use ): 1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
CC
I think this is a point that can be optimized. /cc
I will give a fix for this
Thanks for reporting the issue, this innerSpecName and outerSpecName are confusing and cause issues. Please ping me for review when you have a fix. Thanks!
Please don't triage your own issues immediately, give someone else a chance to review them :( /sig storage
okay, I know


Question: "k8s.io/klog" "k8s.io/kubernetes/pkg/features" schedulerlisters "k8s.io/kubernetes/pkg/scheduler/listers" <ins> "k8s.io/kubernetes/pkg/scheduler/metrics" </ins> schedulernodeinfo "k8s.io/kubernetes/pkg/scheduler/nodeinfo" nodeinfosnapshot "k8s.io/kubernetes/pkg/scheduler/nodeinfo/snapshot" )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a metric to track scheduler cache size. metric name Description -- -- schedulercachesize Number of elements in scheduler cache. Why is this needed: Gives better observability of scheduler cache. For, the number of "assumedpods" gives us an idea of how many pods are "assumed" and therefore being "bound". We can compare this with number of to make sure assume-bind works as expected. /sig scheduling
Is anyone already working on these metrics? If not I'd like to help out /assign
Thanks . Feel free to work on this one. Just a reminder, we may need feedback/approval from sig-instrumentation.


Question: "k8s.io/apimachinery/pkg/types" kwait "k8s.io/apimachinery/pkg/util/wait" cloudprovider "k8s.io/cloud-provider" <ins> volerr "k8s.io/cloud-provider/volume/errors" </ins> "k8s.io/klog" "k8s.io/utils/keymutex" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Pods with PVs could not be moved from one node to another. According to the controller-manager logs, K8s tried to detach to Azure disks from the same node. One of the detach operation waited for this lock (). While it waited for the lock, the controller-manager pod was restarted (for whatever reason, not related to this bug, lets assume an operator restarted the pod). After the restart, K8s only tried to attach the disk to the new node but is not trying to detach it first. What you expected to happen: After a restart of the controller-manager, it should continue unfinished detach operations How to reproduce it (as minimally and precisely as possible): 2 Pods with PVs running on one node Move both pods to another node restart the controller-manager shortly after the move Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.1 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
/area provider/azure that's expected behavior, k8s controller won't continue detach volume if controller manager restarted, user needs to detach that disk volume manually.
Could you please elaborate why this behaviour is expected. I would assume that the controller manager tries to reach the desired state after a restart? Then it would need to detect that the volume is still attached to the wrong node and should try to detach it once more, before it continuing to attach the (still attached) volume? This leads to many ... Why is it required to resolve this situation by manual interaction?
I can't imagine that "manual intervention" is "desired behaviour" (yes, you wrote "expected", but that doesn't matter, if you make no proposal, how to improve this). If this isn't fixed, we can't use Azure, period. We can't have clusters that need to be babysitted / you cannot expect people running workloads on Azure clusters and babysit them and manually fixing what should/can fix itself.

Level-triggered behaviour and resilience is what made Kubernetes so successful... please bring that also to Azure's KCM/CCM.
do you know why controller manager is restarted, it's not a common scenario, we cannot guarantee disk attach/detach functionality works if controller manager restarted at that time.
related to
I think there can be many reasons why the controller-manager is restarted during running detach or attach operations e. g. the hosting machine turn into error state, if you run the controller-manager in k8s -vertical pod autoscaling,... etc. I understand you can't guarantee that attach/detach works when the controller-manager is restarted, but I don't understand why the controller-manager can't recover the situation autonomously after a restart. After restart it tries to attach the volume of the rescheduled pod and this does not work because the volume is still attached to another machine. This is indicated by a error from the arm and the controller-manager could react to it properly, right?
This is an upstream issue, and there is a discussion in , let's check whether we can fix it in cloud provider driver in that thread. Thanks.
I will try to fix this issue in azure disk driver, thanks all for the discussion. details: /assign
Thank you very much It means a lot to us, that there are no manual operations. I wonder how you guys deal with it in AKS, since Azure uses the same controllers, right? Anyway, it helps tremendously, that the clusters are self-healing.
AKS use same kube-controller-manager, while it won't be restarted easily.
OK, but how? I understand that AKS only supports 100 nodes. So, I guess you have some upper limit and just hope, nobody stresses it, because if this happens, you it either restarts or you need to scale it up and then it gets restarted as well. Also, regular maintenance, low-level as well as Kubernetes updates, all must impact it. Considering the probably thousands over thousands of clusters (I don't know, but I expect you run many), any such problem, even if unlikely by itself, must be in your interest to fix, simply because the number of clusters increases the likelihood that it happens, I would think. Anyway, thank you for making this change
Yes, that's reasonable, I will try to fix it as my top priority.


Question: }) Context("CSI volume limit information using mock driver", func() { <del> It("should report attach limit when limit is bigger than 0", func() { </del> <ins> It("should report attach limit when limit is bigger than 0 [Slow]", func() { </ins> // define volume limit to be 2 for this test var err error

created a spread sheet of the slowest running tests for the pull-kubernetes-gce job. cc discussion on slack: spreadsheet: pie chart of test durations from SIGs: ! it seems appropriate to evaluate the state of the tests and reduce these runtimes. possibly move tests to periodic by marking them as [Slow]. /assign /kind design /area test (?)
/remove-kind bug
Are pull jobs the right place to be calculating the time? Or should be be counting in CI instead?
pull is the one timing out, so probably?
Looking at some of the top slowness:: subPath should fail for new directories when readOnly specified in the volumeSource: This test case is no longer defined in master HostPathType: This test case has not been checked in yet and is trying to be in a PR PersistentVolumes-local StatefulSet with pod affinity should use volumes spread across nodes when pod has anti-affinity: This test is already marked as slow in master
Created This should address most of the top slow storage tests


Question: gcePDDetachPollTime = 10 * time.Second nodeStatusTimeout   = 10 * time.Minute nodeStatusPollTime  = 1 * time.Second <ins> podEvictTimeout     = 2 * time.Minute </ins> maxReadRetry        = 3 minNodes            = 2 ) var _ = SIGDescribe("Pod Disks", func() { var ( <ins> ns         string cs         clientset.Interface </ins> podClient  v1core.PodInterface nodeClient v1core.NodeInterface host0Name  types.NodeName

When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .


Question: }) <del> // This test verifies that max-pods flag works as advertised. It assumes that cluster add-on pods stay stable // and cannot be run in parallel with any other test that touches Nodes or Pods. It is so because to check // if max-pods is working we need to fully saturate the cluster and keep it in this state for few seconds. // // Slow PR #13315 (8 min) ginkgo.It("validates MaxPods limit number of pods that are allowed to run [Slow]", func() { totalPodCapacity = 0 for _, node := range nodeList.Items { framework.Logf("Node: %v", node) podCapacity, found := node.Status.Capacity[v1.ResourcePods] framework.ExpectEqual(found, true) totalPodCapacity += podCapacity.Value() } WaitForPodsToBeDeleted(cs) currentlyScheduledPods := WaitForStableCluster(cs, masterNodes) podsNeededForSaturation := int(totalPodCapacity) - currentlyScheduledPods ginkgo.By(fmt.Sprintf("Starting additional %v Pods to fully saturate the cluster max pods and trying to start another one", podsNeededForSaturation)) // As the pods are distributed randomly among nodes, // it can easily happen that all nodes are satured // and there is no need to create additional pods. // StartPods requires at least one pod to replicate. if podsNeededForSaturation > 0 { framework.ExpectNoError(testutils.StartPods(cs, podsNeededForSaturation, ns, "maxp", *initPausePod(f, pausePodConfig{ Name:   "", Labels: map[string]string{"name": ""}, }), true, framework.Logf)) } podName := "additional-pod" WaitForSchedulerAfterAction(f, createPausePodAction(f, pausePodConfig{ Name:   podName, Labels: map[string]string{"name": "additional"}, }), ns, podName, false) verifyResult(cs, podsNeededForSaturation, 1, ns) }) </del> // This test verifies we don't allow scheduling of pods in a way that sum of local ephemeral storage limits of pods is greater than machines capacity. // It assumes that cluster add-on pods stay stable and cannot be run in parallel with any other test that touches Nodes or Pods. // It is so because we need to have precise control on what's running in the cluster.

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-cos-k8sbeta-serial (ci-kubernetes-e2e-gce-cos-k8sbeta-serial) Which test(s) are failing: Various CSI Driver tests. Since when has it been failing: Earliest observed: 11/17 Consistently failing since: 11/29 Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.17 /sig storage /priority critical-urgent
/assign to investigate. are the expansion test flakes supposed to be fixed?
It appears as though some of the failures are also due to tests owned by sig-scheduling: /sig scheduling
The only failures I see in CSI+expansion is from gce-pd CSI driver. We need to fix for that.
We should move "SchedulerPredicates [Serial] validates MaxPods limit number of pods that are allowed to run [Slow]" out of e2e tests. It relies on a particular state of the cluster before the test start that is hard to validate. It should be more than enough to have an integration test for this, even a unit test (which we already hav ) should be fine .
will you be willing to work on that? it's flaking A LOT right now, definitely more than 50%
I created , should be addressed soon.
There are several tests flaky. Summarize them as below expansion test See comments from and a few other tests The cause of test failure is node is OutOfPod resource. See one of the log message {kubelet test- This is due to the SchedulerPredicates test mentioned above , which creates more than 100 Pods at the same time. We need to move this test out to make sure other tests have enough resources.
I assume this was not cherry-picked to 1.17, right?
/reopen this is still an issue on 1.17. we should probably consider cherry-picking as this is creating a lot of noise in the dashboards
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/kind flake
correct, it was not cherry-picked to 1.17.
/retitle [Flaking Test] gce-cos-k8sbeta-serial (ci-kubernetes-e2e-gce-cos-k8sbeta-serial)
/retitle [Flaky Test] gce-cos-k8sbeta-serial (ci-kubernetes-e2e-gce-cos-k8sbeta-serial)
/milestone v1.18
/cc
/assign Based on 's analysis earlier, the test failures are due to a scheduler test taking up too many resources. I believe that scheduler test fix needs to be cherry-picked. Assigning to sig-scheduler to track it. /remove-sig storage
/unassign
Those labels are not set on the issue: <detailsIn response to : /assign Based on 's analysis earlier, the test failures are due to a scheduler test taking up too many resources. I believe that scheduler test needs to be cherry-picked. Assigning to sig-scheduler to track it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Those labels are not set on the issue: <detailsIn response to : /assign Based on 's analysis earlier, the test failures are due to a scheduler test taking up too many resources. I believe that scheduler test fix needs to be cherry-picked. Assigning to sig-scheduler to track it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.17
/retitle [Failing Test] gce-cos-k8sbeta-serial (ci-kubernetes-e2e-gce-cos-k8sbeta-serial) /remove-kind flake /remove-sig scheduling /sig storage
/unassign /assign Looks like the cherry pick did not cause this to bounce back. Any ideas why that would be? cc
/milestone v1.18
Is this failure specifically for 1.17 (because we only cherry picked into 1.18)?
Looks like the 1.17 serial test grid is looking fine?

(slightly flaky)
Can we also use a new bug to track the quay related failures which are unrelated to the original issue?
This job is now green


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package fake import ( "bytes" "context" "io" "testing" corev1 "k8s.io/api/core/v1" cgtesting "k8s.io/client-go/testing" ) func TestFakePodsGetLogs(t *testing.T) { fp := FakePods{ Fake: &FakeCoreV1{Fake: &cgtesting.Fake{}}, ns:   "default", } req := fp.GetLogs("foo", &corev1.PodLogOptions{}) body, err := req.Stream(context.Background()) if err != nil { t.Fatal("Stream pod logs:", err) } var buf bytes.Buffer n, err := io.Copy(&buf, body) if err != nil { t.Fatal("Read pod logs:", err) } if n == 0 { t.Fatal("Empty log") } err = body.Close() if err != nil { t.Fatal("Close response body:", err) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I tried to use  in a unit test, however I get a nil pointer panic when is invoked. What you expected to happen: would return a that doesn't panic on any method call. Ideally, it would allow me to interact with the client in someway for testing. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: client-go version: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
I believe this also impacts returning logs via a reactor as does not implement
Seems to be a problem with . Since we're creating the request with all empty fields. Not sure what the best way it to resolve this. The best thing I came up with is to use to return a request with everything set up. But I'm a noob here so that's just a shot in the dark :man_shrugging:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
The request returned has a nil which is used in the
I have filed a PR trying to fix this issue.


Question: // namespaces if it's namespace scoped, so leave // APIResource.Namespaced as false is all right. apiResource := unversioned.APIResource{Name: resource.Resource} <del> // The default parameter codec used by the dynamic client cannot // encode api.ListOptions. // TODO: api.ParameterCodec doesn't support thirdparty objects. // We need a generic parameter codec. return client.ParameterCodec(api.ParameterCodec).Resource(&apiResource, api.NamespaceAll).List(&options) </del> <ins> return client.ParameterCodec(dynamic.VersionedParameterEncoderWithV1Fallback). Resource(&apiResource, api.NamespaceAll). List(&options) </ins> }, WatchFunc: func(options api.ListOptions) (watch.Interface, error) { // APIResource.Kind is not used by the dynamic client, so

It seems to me like all of the garbage collection integration tests produce a ton of log messages of the form:


Question: library = ":go_default_library", deps = [ "//vendor/k8s.io/api/admissionregistration/v1alpha1:go_default_library", <ins> "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/meta:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/runtime:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", <ins> "//vendor/k8s.io/apiserver/pkg/admission:go_default_library", "//vendor/k8s.io/apiserver/pkg/authorization/authorizer:go_default_library", </ins> ], )

There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery