Question: } } <ins> func NewRevisionDeletePodFailure(t *testing.T, set *apps.StatefulSet, invariants invariantFunc) { client := fake.NewSimpleClientset(set) om, _, ssc := setupController(client) if err := scaleUpStatefulSetControl(set, ssc, om, invariants); err != nil { t.Errorf("Failed to turn up StatefulSet : %s", err) } var err error set, err = om.setsLister.StatefulSets(set.Namespace).Get(set.Name) if err != nil { t.Fatalf("Error getting updated StatefulSet: %v", err) } if set.Status.Replicas != 3 { t.Error("Failed to scale StatefulSet to 3 replicas") } selector, err := metav1.LabelSelectorAsSelector(set.Spec.Selector) if err != nil { t.Error(err) } pods, err := om.podsLister.Pods(set.Namespace).List(selector) if err != nil { t.Error(err) } // trigger a new revision updateSet := set.DeepCopy() updateSet.Spec.Template.Spec.Containers[0].Image = "nginx-new" if err := om.setsIndexer.Update(updateSet); err != nil { t.Error("Failed to update StatefulSet") } set, err = om.setsLister.StatefulSets(set.Namespace).Get(set.Name) if err != nil { t.Fatalf("Error getting updated StatefulSet: %v", err) } // delete fails om.SetDeleteStatefulPodError(apierrors.NewInternalError(errors.New("API server failed")), 0) _, err = ssc.UpdateStatefulSet(context.TODO(), set, pods) if err == nil { t.Error("Expected err in update StatefulSet when deleting a pod") } set, err = om.setsLister.StatefulSets(set.Namespace).Get(set.Name) if err != nil { t.Fatalf("Error getting updated StatefulSet: %v", err) } if err := invariants(set, om); err != nil { t.Error(err) } if set.Status.CurrentReplicas != 3 { t.Fatalf("Failed pod deletion should not update CurrentReplicas: want 3, got %d", set.Status.CurrentReplicas) } if set.Status.CurrentRevision == set.Status.UpdateRevision { t.Error("Failed to create new revision") } // delete works om.SetDeleteStatefulPodError(nil, 0) status, err := ssc.UpdateStatefulSet(context.TODO(), set, pods) if err != nil { t.Fatalf("Unexpected err in update StatefulSet: %v", err) } if status.CurrentReplicas != 2 { t.Fatalf("Pod deletion should update CurrentReplicas: want 2, got %d", status.CurrentReplicas) } if err := invariants(set, om); err != nil { t.Error(err) } } </ins> func emptyInvariants(set *apps.StatefulSet, om *fakeObjectManager) error { return nil }

It was found during a review that stateful set status might not be correctly updated when new revision is being rolled out and API error occurs when deleting a pod. The status should be updated correctly Not easy to reproduce, but should be possible with this unit test No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps


Question: [Install] WantedBy=multi-user.target <ins> - path: /etc/modprobe.d/sunrpc.conf permissions: 0644 owner: root # The GKE metadata server uses ports 987-989, so the sunrpc range should be restricted to be below. content: | options sunrpc max_resvport=986 </ins> runcmd: - systemctl daemon-reload - systemctl enable kube-node-installation.service

When mounting an in-tree NFS share, reserved ports are used (this happened to be a filestore instance, so we can assume the server is configured sanely). gke-regular-default-pool--zgfs /home/mattcary # netstat -tp | grep nfs tcp 0 0 gke-regular-default:982 10.108.60.194:nfs ESTABLISHED - Note the connection is at port 982. This can conflict with other ports on the host, eg the metadata server in GKE. The right solution seems to be to restrict the sunrpc reserved port range. This only affects kube-up.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig storage


Question: allErrs = append(allErrs, field.Required(fldPath.Child("resources").Key(string(core.ResourceStorage)), "")) } else { allErrs = append(allErrs, ValidateResourceQuantityValue(string(core.ResourceStorage), storageValue, fldPath.Child("resources").Key(string(core.ResourceStorage)))...) <del> if storageValue.Value() == int64(0) { allErrs = append(allErrs, field.Invalid(fldPath, storageValue, isZeroErrorMsg)) } </del> <ins> allErrs = append(allErrs, ValidatePositiveQuantityValue(storageValue, fldPath.Child("resources").Key(string(core.ResourceStorage)))...) </ins> } if spec.StorageClassName != nil && len(*spec.StorageClassName) > 0 {

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT /kind bug What happened: If someone requests a zero (0) capacity PVC and an administrator creates a zero (0) capacity PV, Kubernetes will happily bind them together. Unfortunately, any use of this PVC/PV combination by a pod will fail. It fails because the zero capacity PV cannot be formatted or mounted successfully. Note: The same issue happens for dynamically provisioned PVs. What you expected to happen: I expect that Kubernetes would not allow creation of invalid PVC and PV objects, aka zero (0) capacity PVs and PVCs. How to reproduce it (as minimally and precisely as possible): Create a pod that requests a zero capacity PVC, using dynamic PV provisioning is simpler than creating your own zero (0) capacity PV. The pod will fail to reach a running state, since the related PV fails to be formatted and mounted successfully. Here is the error that I received for a test I performed requesting a zero capacity PVC using ceph rbd: Anything else we need to know?: Here is a related item with a specific fix for external ceph rbd provisioner: Environment: Nothing specific to environment
/sig storage
/assign


Question: package kubectl import ( <ins> "bytes" </ins> "fmt" "reflect" "strings"

Kubectl describe pods id only has top level pod information, without any information related to containers. Taking kube-dns pod as an example, which has 3 containers specified: If a pod with RestartAlways policy, but all contianers are in crashloop, the user cannot interpret the true state based on above information. Proposed to include ContainerStatus here.
Why isn't this redundant with ?
is for kubectl get pods id.


Question: # Get default service account credentials of the VM. GCE_METADATA_INTERNAL="http://metadata.google.internal/computeMetadata/v1/instance" function get-credentials { <del> curl --fail --retry 5 --retry-delay 3 --retry-connrefused --silent --show-error "${GCE_METADATA_INTERNAL}/service-accounts/default/token" -H "Metadata-Flavor: Google" -s | python3 -c  </del> <ins> curl --fail --retry 5 --retry-delay 3 --retry-connrefused --silent --show-error "${GCE_METADATA_INTERNAL}/service-accounts/default/token" -H "Metadata-Flavor: Google" -s | ${PYTHON} -c  </ins> 'import sys; import json; print(json.loads(sys.stdin.read())["access_token"])' }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gce-device-plugin-gpu Cluster up Apr 23 8:30PM PDT Taken from: From a , there are supposed to be three worker/minion nodes set up as well. Started failing after -- so maybe related to ? /cc /sig testing More examples of failing runs: /priority critical-urgent (this is a job on the release/master-blocking board)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It uses image gke-1134-gke-rc5-cos-69--138-0-v190320-pre-nvda-gpu as node image kubekins-e2e image is built from Base image is bootstrap image is built from with debian:buster and install python3.
The node image gke-1134-gke-rc5-cos-69--138-0-v190320-pre-nvda-gpu has no python3. We need to upgrade it.
Thanks for catching this and sending revert. Separately, we should definitely upgrade that image, cos-69 is super old and not supported anymore: /cc regarding GPU image


Question: - gocritic text: "ifElseChain: rewrite if-else to switch statement" <ins> # Only packages listed here opt into the strict "exported symbols must be documented". # # Exclude texts from https://github.com/golangci/golangci-lint/blob/ab3c3cd69e602ff53bb4c3e2c188f0caeb80305d/pkg/config/issues.go#L11-L103 - linters: - golint - revive - stylecheck text: comment on exported (method|function|type|const)|should have( a package)? comment|comment should be of the form|comment on exported (method|function|type|const)|should have( a package)? comment|comment should be of the form|exported (.+) should have comment( (or a comment on this block))? or be unexported|package comment should be of the form "(.+)...|comment on exported (.+) should be of the form "(.+)...|should have a package comment path-except: cmd/kubeadm </ins> linters: disable-all: false enable: # please keep this alphabetized

Looks like the golangci-lint-pr-hints linter was updated recently to put the feature gate comment ahead of the owner field in All the feature gates in are ordered by the following comments: // owner: // kep: // alpha: // beta: // // Describe the feature gate <declare the feature gateThe new linter changes now requires that the comment describing the feature gate should even precede the owner comment line which goes against the structure followed by the entire file and this needs to be fixed Feature gates should be ordered as follows to maintain parity with the rest of // owner: // kep: // alpha: // beta: // // Describe the feature gate <declare the feature gateUpdate/add new feature gate to No response <details</detailsmaster branch of k8s/k8s <details</detailsN/A <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
!
cc
cc
it was supposed to only apply to cmd/kubeadm.
/sig testing
/assign
Thank you for fixing that.
If I break it, I fix it... :hot_face:


Question: package app <del> func initForOS(service bool) error { </del> <ins> func initForOS(service bool, priorityClass string) error { </ins> return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Windows nodes running with high (90%) CPU usage are having calls to /stats/summary time out with "Context Deadline Exceeded". When this happens metrics-server cannot gather accurate CPU/memory usage for the node or pods running on the node which results in HPA not functioning returning <unknowninstead of values What you expected to happen: Windows nodes should be able to run with high CPU utilization without negatively impacting Kubernetes functionality How to reproduce it (as minimally and precisely as possible): a cluster with Windows Server 2019 nodes CPU intensive workloads into cluster ex: workload to put node into high CPU state node metrics with Anything else we need to know?: This issue appears to be easier to reproduce on 1.18.x nodes vs 1.16.x or 1.17.x nodes Ex: ! ! metrics-server logs show E1020 10:58:24. 1 ] unable to fully collect metrics: unable to fully scrape metrics from source kubeletsummary:akswin000000: unable to fetch metrics from Kubelet akswin000000 (10.240.0.115): Get https://10.240.0.115:/stats/summary?onlycpuandmemory=true: context deadline exceeded Kubelet logs show and Environment: Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: AKS / Azure OS (e.g: ): WIndows Server 2019 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: Docker EE version 19.3.11
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig windows /area kubelet
I was able to reproduce on 1.18.8 during a scale operation: where it fully subscribes all CPU's in Allocatable: Metric server logs: I do not see "context deadline" in kubelet but I did find some logs assocated: Once the pods are scheduled the node remains at 100% cpu and I no longer see errors but is not accurate: on the node (run after the pod deployment stabilized)
Adding to kubelete's flag and I no longer see the errors. This reserves some cpu for kubelet and other processes. We call this out in the but do not do the same for cpu.
Also, without setting system reserved CPU, it looks like does not accurately reflect CPU usage on the nodes when CPU utilization is very high. When I got some nodes into this state I did not see any error messages in logs from metrics-server and also I noticed that that querying the metrics API directly from showed recent timestamps.
I run into this again during scaling, collectiong logs for analysis here: Metric Server logs: Kubelet logs:
have you experienced this at all?
I believe has seen high CPU issues before. Not particularly on this. I've heard there were kernel issues in the past that triggered high CPU cases.
Some investigations on our end are showing that the DiagTrack service running in the infra/pause container can consume non-trivial amounts of CPU. I'll create an issue to address that problem as it might help out some here. That certainly will not eliminate the problem we are seeing tho.
is an issue for DiagTrack
The differences in behavior between 1.17. and 1.18. clusters looks like it is caused by (which was NOT backported to any prior releases) K8s version <= 1.17 behavior Prior to these changes CPU weights for containers were inferred based on limit values set in the pod specs. The observed behavior on Windows system when CPU weights were used was that under high-CPU load pods/containers were not guaranteed their scheduling time. The good news here is that system critical services like kubelet/docker/etc were still able to get enough CPU time for the nodes to function properly. K8s version = 1.18 behavior After these changes CPU limits were enforced for pods.

The observed behavior on 1.18 clusters is the when a Windows node enters a high-CPU state is that containers are guaranteed their scheduling time but host resources start to get starved. Usually this manifests as the node entering the state, metrics not working correctly (because calls to /stats/summary are timing out) or both. Suggestion / possible fixes Set the Windows process priority of the kubelet to  to help ensure it gets CPU time Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) Set reserve CPU with with --system-reserves=[cpu=500m] This needs to scaled based on number of CPUs on host based on my experimentations Currently guidance on for Windows only mentions reserving system-memory It is still possible to over-commit CPU and starve kubelet/docker/HCS if users schedule pods with mis-matching requests and limits Update documentation.
Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) I can work on this, I am wondering if we should do the same for CRI(like docker) and CNI processes.
Sure, if you want to work on this go ahead. I think we should probably try to make this be toggleable via a kubelet flag. I believe there are other windows specific flags already implemented in the kubelet (like one to run as a windows service IIRC)
docker/containerd should probably have a flag that allows them to run as high priory as well but would likely be different issues in the respective projects
/reopen I don't think this was ever fully addressed
Reopened this issue. <detailsIn response to : /reopen I don't think this was ever fully addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
also related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.Logf("Waiting for the ReplicaSet to have the right controllerRef") err = checkDeploymentReplicaSetsControllerRef(c, ns, deploy.UID, podLabels) Expect(err).NotTo(HaveOccurred()) <ins> framework.Logf("Verifying no extra ReplicaSet is created (Deployment %q still has only one ReplicaSet after adoption)", deploymentName) rsList = listDeploymentReplicaSets(c, ns, podLabels) Expect(len(rsList.Items)).Should(Equal(1)) framework.Logf("Verifying the ReplicaSet has the same UID as the orphaned ReplicaSet") Expect(rsList.Items[0].UID).Should(Equal(orphanedRSUID)) </ins> } func checkDeploymentReplicaSetsControllerRef(c clientset.Interface, ns string, uid types.UID, label map[string]string) error {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Deployment's collision avoidance mechanism assumes that if a replicaset's PodTemplateSpec is semantically deep equal with the deployment's PodTemplateSpec, then the replicaset is the new replicaset for the deployment. This assumption does not take into account the case when the user updates only a field of the replicaset that does not reside within the replicaset's PodTemplateSpec, in which the collision avoidance mechanism should also be triggered. What you expected to happen: The collision avoidance mechanism should be triggered when a non-PodTemplateSpec field (e.g., field) of the replicaset is changed. After making changes to (), the e2e test can be elaborated () to verify the mechanism works as expected. How to reproduce it (as minimally and precisely as possible): a Deployment for ReplicaSet and its Pod(s) to become ready a non-PodTemplateSpec field of the ReplicaSet that Deployment does not create a new ReplicaSet /sig apps /cc
To clarify, we don't actually care about RS label changes in this case. What we do care about is the change of ownership between Deployment and its RSes.
Update description of the issue. Tell me if it is still confusing.
Thanks for pointing to the e2e test.


Question: "gke_environment_test.go", "hugepages_test.go", "image_id_test.go", <del> "kubelet_test.go", </del> "log_path_test.go", "mirror_pod_test.go", "node_container_manager_test.go",

What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. To my eyes they may meet the criteria for conformance. There's definitely nothing about them that is node specific. They should be running as part of our regular CI, and they should probably be running as presubmits. Porting them to test/e2e/common would cause this to happen. /kind cleanup /sig node /area test /assign
GitHub didn't allow me to assign the following users: mkurylec. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. /sig node /area test /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } if resource.Resource == item.ShortForm.Resource { resource.Resource = item.LongForm.Resource <ins> resource.Group = item.LongForm.Group </ins> return resource } }

Copying the issue from Here is my CRD CRD Data Outputs Expected to see the resource list when kubectl get dpl Versions
/sig api-machinery /area custom-resources /kind bug
Can you check if the title is right for this case?
/cc
If you collide with another short name, you have to qualify with a group. It should follows normal resolution rules for priority
this isn't CRD-specific. isn't a colliding shortname. it expanded dpl -deployments, then resolved deployments against preferred group order, instead of limiting resolution to the group(s) that had the shortname
updated description to narrow the report to the case which is a bug
Did a little research, turn out kubectl will take as it's short name: I can give a shot to solve this problem, unless it's by design... : )
FYI. PR .
/assign


Question: golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k= golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= <del> golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= </del> golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= <del> golang.org/x/tools v0.12.0/go.mod h1:Sc0INKfu04TlqNoRA1hgpFZbhYXHPr4V5DzpSBTPqQM= </del> <ins> golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= </ins> golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=

Drop the and dependencies from to keep the import lighter. /sig auth /triage accepted
/assign Hi, , I created a PR to fix this. I am not sure If I am correct due to the repo seems generated. Would you like to provides some hints for me?
This will likely require a new go module in to move the client-go dep and then mark it as internal, ex: . is the only API machinery dep so we can probably come up with some way to re-implement the wait logic locally? Then we drop these lines so that these imports do not come back again:
Whoa, yeah, I'd take a for loop and a time.Sleep in a test to drop those deps
What is kms using client-go for? Tests only?
These were all needed by the KMS repo back when we had a much more complicated reference implementation which required a key hierarchy. Now they are just used in some tests and mock implementations, so we should be able to simplify things.
I found a place for rate limit. #liggitt
that package just delegates to "", switch that mock provider to use that directly?


Question: }) ginkgo.By("Giving the Kubelet time to start up and produce metrics") <del> gomega.Eventually(ctx, getKubeletMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) </del> <ins> gomega.Eventually(ctx, getKubeletMetrics, 2*time.Minute, 10*time.Second).Should(matchResourceMetrics) </ins> ginkgo.By("Ensuring the metrics match the expectations a few more times") <del> gomega.Consistently(ctx, getKubeletMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) </del> <ins> gomega.Consistently(ctx, getKubeletMetrics, 2*time.Minute, 10*time.Second).Should(matchResourceMetrics) </ins> }) }) })

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node started to fail since 09-13.
/cc
/assign
/cc
/triage accepted /priority important-soon it's indeed failing too often
/reopen The test is still failing: Now timing out at 120s i.e. 2 minutes as we increased the default timeout . We need to understand the root cause and get to the bottom of this.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Snippet from the build log () corresponding from this failure We see from the above logs that: node has two numa nodes, 8 cpus per NUMA so the root cause of the failure is not because the tests are running on a node that doesn't have multi-numa support. is started and the health check fails a couple of times but eventually succeeds. pod is created with 9 cpus (cpusperNUMA +1 i.e. 8+1) which is expected to fail with Topology Affinity Error as we cannot allocate resources for such a pod from a single NUMAnode. We fail to get matching metrics. It is not clear if the check fails because we don't get kubelet metrics at all or the expected metrics but based on point 6 below it seems like it is because of latter. We can add additional logs for more clarity here. each is executed and Kubelet is restarted and health check is done. are collected from the namespace corresponding to the test pod which shows that the pod started even though it was expected to fail with Topology Affinity Error. We need to understand why the pod got successfully created even though it requested CPUs that couldn't be allocated from a single NUMAnode.


Question: claim := obj.(*v1.PersistentVolumeClaim) reactor.DeleteClaimEvent(claim) // wait until claim is cleared from cache, i.e., deleteClaim is called <del> for len(ctrl.claims.ListKeys()) > 0 { time.Sleep(10 * time.Millisecond) </del> <ins> err := wait.Poll(10*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { return len(ctrl.claims.ListKeys()) == 0, nil }) if err != nil { return err </ins> } // make sure the operation timestamp cache is NOT empty if !ctrl.operationTimestamps.Has("volume5-6") {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are some places in that wait for conditions . We should update all the places to set something like /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --What happened: There are some places in that wait for conditions . We should update all the places to set something like /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup /remove-kind bug
I can try to fix this :) /assign
/sig storage


Question: - -k - --cache-size=1000 - --no-negcache <ins> - --dns-loop-detect </ins> - --log-facility=- - --server=/{{ .DNSDomain }}/{{ .DNSBindAddr }}#10053 - --server=/in-addr.arpa/{{ .DNSBindAddr }}#10053

Is this a BUG REPORT or FEATURE REQUEST?: What happened: If hosts points to a local DNS server, or if it contains the cluster's DNS server (to enable looking up services from host), it'll cause DNS loop which can effectively disable kube-dns or make it very slow. What you expected to happen: At least, kube-dns should detect localhost or cluster DNS server entries in and not send queries to them. How to reproduce it (as minimally and precisely as possible): Use a system with local cashing DNS server, or add cluster's DNS server address to hosts Anything else we need to know?: According to documentation, adding should make dnsmasq detect that and disable the upstream DNS server which cause the loop. It should prevent DNS from becoming unresponsive in the first case (a better solution is to extract real upstream servers, which probably cannot be done for every configuration); and for the second case it should skip sending queries to itself and send to real upstream servers.
/sig network
Could you describe the use-case more? If the host's contains cluster DNS server and some other upstream server - like 8.8.8.8, will services lookups work reliably from the host? If the services lookup goes to the 8.8.8.8 server, it will return NXDOMAIN. If it hits the clusterDNS server, only then it would work, correct?
Also ref If the first server returns NXDOMAIN, the second server will not be tried. How do you make both cluster DNS name and non-cluster DNS name resolvable on the host?
If cluster DNS server comes before a real DNS server, it works fine. Also note that what man page says is the glibc (or libc) behavior, not other users (e.g. dnsmasq sends its queries to any one of them, but can be configured to query them in strict order ( flag), or to query all of them (). Anyway, consider when cluster DNS server comes before the real server: cluster DNS queries, the cluster DNS server responds with the IP of the cluster or pod. external names, the query is sent to cluster DNS server. It doesn't find the name, so it tries to send it to upstream DNS servers; which in my example are cluster DNS server itself and an external DNS server.

If dnsmasq sends it to external DNS server, it'll receive the answer and can serve the query. If it sends it to itself, there is a loop and the same process happens again until dnsmasq decides to query the other server (which might not happen!). WIth flag, when dnsmasq sends the query to itself, it'll recognize the loop and will remove its own address from the list of upstream DNS servers, so it'll always send external names to the external DNS server from now on.
Thanks for the explanation. I believe it might work with explicit configuration. Though adding a flag to upstream for fulfilling a not quite common use case seems a bit odd to me. Have you thought of configuring upstream DNS servers via kube-dns config [1] such that it will not send request to itself at all? [1] -
Note that the flag has two use cases, the other one is prevent kube-dns breakage when upstream DNS server is a local one (e.g. If you deploy to Ubuntu, which has local caching DNS server by default (e.g. in 18.04, it is systemd-resolved and contains 127.0.0.53)). Actually I guess there are a number of DNS related issues which might be solved by this flag too. So, while this flag will not make kubernetes magically find real upstream dns servers in such cases, it prevents kube-dns crash; which has caused frustration and some time for me to discover. So, either kubelet should error out and exit if the given contains a local address, or such flag should be used by kube-dns to prevent DNS loops. And, IIRC, the only reason that cluster's DNS server is not to host, at least by kubespray, is mentioned to be host specific in the docs. So, my use case seems to be an ideal one and it would be the default one if it was easy to setup for different platforms. About using kube-dns config, no. We're going to provide automatic kubernetes deployment for many environments, and so we don't know the upstream DNS servers upfront, and we don't want to hard code it. We want it to be the DNS servers used in the target network. So, configuring a specific upstream DNS server is a no-go.


Question: "fmt" "net" "path" <del> "runtime" </del> kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm" "k8s.io/kubernetes/cmd/kubeadm/app/images"

Right now we create 3 daemonsets for in , but will only give use image names for , so this feature is broken. We should only deploy one daemon set for the native architecture, but ensure it has the right label selectors, so we don't try to run pods on nodes of different architecture. We should also make sure all addons have architecture labels, and not just . If user is building heterogeneous clusters, the will need to take about deploying addons for additional architectures, we shouldn't pre-deploy multiple things knowing that they will most likely remain unschedulable.
I have the same issue as well. I tried to create a cluster with x86 (Intel NUC) and arm (Raspberry-Pi) and I see that the kube-proxy is crashing. Here's the log entry from kube-proxy Logs from kube-proxy exec user process caused "exec format error" Attached is the kube-proxy yaml file. I see that its picking up arm image Don't know how to fix this.


Question: stopCh := make(chan struct{}) go podInformer.Run(stopCh) <ins> // wait for the podInformer to observe the pod, otherwise the rc manager // will try to create a new pod rather than adopting the existing one. if err := wait.Poll(10*time.Second, 60*time.Second, func() (bool, error) { objects := podInformer.GetIndexer().List() for _, object := range objects { pod, ok := object.(*api.Pod) if !ok { t.Fatal("expect object to be a pod") } if pod.Name == podName { return true, nil } } return false, nil }); err != nil { t.Fatal(err) } </ins> go rm.Run(5, stopCh) if err := wait.Poll(10*time.Second, 60*time.Second, func() (bool, error) { updatedPod, err := podClient.Get(pod.Name)

https://k8s- Failed: TestAdoption {integration}
https://k8s- Failed: TestAdoption {integration}
https://k8s- Failed: TestAdoption {integration}
https://k8s- Failed: TestAdoption {integration}
The problem is that the rc manager hasn't observed pod1, so it creates another pod and scales down, pod1 might get deleted. I'm working on the fix.


Question: func main() { flag.Parse() if len(flag.Args()) < 1 { <del> fmt.Fprintf(os.Stderr, "USAGE: %s <DIR or FILE> [...]n", os.Args[0]) </del> <ins> fmt.Fprintf(os.Stderr, "USAGE: %s <DIR or FILE or '-'> [...]n", os.Args[0]) </ins> os.Exit(64) } stableMetrics := []metric{} errors := []error{} <ins> addStdin := false </ins> for _, arg := range flag.Args() { <ins> if arg == "-" { addStdin = true continue } </ins> ms, es := searchPathForStableMetrics(arg) stableMetrics = append(stableMetrics, ms...) errors = append(errors, es...) } <ins> if addStdin { scanner := bufio.NewScanner(os.Stdin) scanner.Split(bufio.ScanLines) for scanner.Scan() { arg := scanner.Text() ms, es := searchPathForStableMetrics(arg) stableMetrics = append(stableMetrics, ms...) errors = append(errors, es...) } } </ins> for _, err := range errors { fmt.Fprintf(os.Stderr, "%sn", err) }

/assign /cc /sig instrumentation
GitHub didn't allow me to assign the following users: serathius. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc /sig instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey Looks like this test is passing on testgrid, https://k8s- Which would point to problem being related to specific environment. From your log looks like problem is specific to Macs. I will try to reproduce it to confirm it. My current ideas for solution are: Disabling test on Mac Instead of passing files as arguments, pass them in STDIN WDYT?
FWIW ARG_MAX can technically vary even on linux ...? STDIN SGTM


Question: if err := jm.podControl.CreatePodsWithControllerRef(job.Namespace, &job.Spec.Template, job, newControllerRef(job)); err != nil { defer utilruntime.HandleError(err) // Decrement the expected number of creates because the informer won't observe this pod <ins> glog.V(2).Infof("Failed creation, decrementing expectations for job %q/%q", job.Namespace, job.Name) </ins> jm.expectations.CreationObserved(jobKey) activeLock.Lock() active-- activeLock.Unlock() <ins> errCh <- err </ins> } }() } wait.Wait() } <del> return active </del> <ins> select { case err := <-errCh: // all errors have been reported before, we only need to inform the controller that there was an error and it should re-try this job once more next time. if err != nil { return active, err } default: } return active, nil </ins> } func (jm *JobController) updateJobStatus(job *batch.Job) error {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? Job, resourcequota, scheduler Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-14T13:36:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: CentOS bare metal () Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Linux <server name3.10.0-514.2.2.el7.x8664 SMP Tue Dec 6 23:06:41 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: - Others: - What happened: Having created a resourcequota (e.g. 400M) and creating enough memory-limited jobs to go over that quota (e.g. 4 x 200M limit jobs), at first, two jobs are created and executed as expected, and two jobs fail at first due to memory quota: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 4s 4s 1 job-controller Warning FailedCreate Error creating: pods "memhog-y152c55u-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi After the first two jobs have been completed, instead of starting the remaining jobs as soon as possible, the remaining two jobs stay unstarted in this state for many, many hours until they finally start and complete.

First job: Start Time: Fri, 28 Apr 2017 15:00:08 +0300 One of the remaining 2 jobs: Start Time: Sat, 29 Apr 2017 07:13:34 +0300 What you expected to happen: I'm expecting the two waiting jobs to start executing as soon as the resources they need are available, e.g. as soon as the first jobs are completed. How to reproduce it (as minimally and precisely as possible): Bring up a kubernetes cluster, then execute these commands: $ kubectl create quota memoryquota --hard=memory=400Mi (cat /dev/urandom head -n 1)' --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory='2'm run- $ chmod +x run- $ for i in {1.4}; do ./run- 200 100; done job "memhog-0ju2eiuy" created job "memhog-t8qyt5sj" created job "memhog-ljaqz4ek" created job "memhog-eb8q8kr3" created Wait for around 30 seconds and the first jobs should be complete: $ kubectl get jobs NAME DESIRED SUCCESSFUL AGE memhog-0ju2eiuy 1 1 43s memhog-eb8q8kr3 1 0 42s memhog-ljaqz4ek 1 0 43s memhog-t8qyt5sj 1 1 43s $ kubectl describe jobs . However, the remaining jobs don't start at all as shown by: $ kubectl describe job memhog-ljaqz4ek Name: memhog-ljaqz4ek Namespace: test Selector: controller-uid=-2f00-11e7-8766- Labels: run=memhog-ljaqz4ek Annotations: run memhog-ljaqz4ek memhog -r100 100m --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory=200Mi,cpu=100m --record=true --image=derekway.

Parallelism: 1 Completions: 1 Pods Statuses: 0 Running / 0 Succeeded / 0 Failed Pod Template: Labels: controller-uid=-2f00-11e7-8766- job-name=memhog-ljaqz4ek run=memhog-ljaqz4ek Containers: memhog-ljaqz4ek: Image: derekwaynecarr/memhog Port: Command: memhog -r100 100m Limits: cpu: 100m memory: 200Mi Environment: <noneMounts: <noneVolumes: <noneEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 job-controller Warning FailedCreate Error creating: pods "memhog-ljaqz4ek-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi Furthermore, pods related to the remaining jobs aren't even created and only the completed pods are shown: $ kubectl get pods -a NAME READY STATUS RESTARTS AGE memhog-0ju2eiuy-bz0zm 0/1 Completed 0 7m memhog-t8qyt5sj-q9msw 0/1 Completed 0 7m Anything else we need to know: I've also created a stackoverflow question about this (), and asked about this in the kubernetes slack, where I was urged to create a bug report.
I will try to reproduce. The creation of a pod will be blocked by quota. In the absence of a pod, the scheduler has nothing to schedule, so it is not a scheduling bug. It's possible the issue is related to back-off in the job controller not attempting to create pods again. The event shows that it had been 3m since the job controller attempted to create pods again for the requisite job which implies some type of back-off. None of the controllers are quota aware, so I would anticipate they should just keep trying at reasonable intervals at this point. If you describe quota after the first 2 pods are completed, do you see quota usage being replenished as expected?
I would like to fix the bug, could you assigin it to me?

if pod create failed, jobcontroller.managejob() need return error, will requeue podcreate task.
-- maybe you have more insight, but it looks to me just reading the code that when a job controller is unable to create a pod due to lack of quota, its not actually an error that forces a re-queue of the job. in addition, i cant actually see when a full resync of the job work queue happens outside of the informer sync interval which is probably really long. see: see:
- feel free to open a pr, please cc me on your solution. i would like to understand from the job folks if there should be a periodic resync interval aside from the need to requeue the job when all pods cannot be spawned (which i assume your pr will handle via exponential backoff)
-- we will want to have a fix for this downstream as well.
Yes, after the first two jobs are complete, shows that 0M out of 400M is in use. I'm also able to create new jobs, up to the memory limit so 2 at a time, that execute as expected. Any jobs over that at a time will stay in the same waiting state.
- Yes, I want handle it via exponential backoff. IMO, MaxOfRateLimiter had implemented a periodic resync, we don't need another one.
I'll make sure to cherry-pick this to openshift ( fyi).


Question: ec.InvalidateCachedPredicateItem(nodeName, invalidPredicates) } <del> // getHashEquivalencePod returns the hash of equivalence pod. // 1. equivalenceHash // 2. if equivalence hash is valid func (ec *EquivalenceCache) getHashEquivalencePod(pod *v1.Pod) (uint64, bool) { </del> <ins> // equivalenceClassInfo holds equivalence hash which is used for checking equivalence cache. // We will pass this to podFitsOnNode to ensure equivalence hash is only calculated per schedule. type equivalenceClassInfo struct { // Equivalence hash. hash uint64 } // getEquivalenceClassInfo returns the equivalence class of given pod. func (ec *EquivalenceCache) getEquivalenceClassInfo(pod *v1.Pod) *equivalenceClassInfo { </ins> equivalencePod := ec.getEquivalencePod(pod) if equivalencePod != nil { hash := fnv.New32a() hashutil.DeepHashObject(hash, equivalencePod) <del> return uint64(hash.Sum32()), true </del> <ins> return &equivalenceClassInfo{ hash: uint64(hash.Sum32()), } </ins> } <del> return 0, false </del> <ins> return nil </ins> }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Today, . This seems unnecessary and should be done once for all the nodes. Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig scheduling cc:
/assign Thanks, will send fix per slack discussion.
/cc


Question: if m.adapter.cgroupManagerType == libcontainerSystemd { updateSystemdCgroupInfo(libcontainerCgroupConfig, cgroupConfig.Name) } else { <del> if libcontainercgroups.IsCgroup2UnifiedMode() { libcontainerCgroupConfig.Path = m.buildCgroupUnifiedPath(cgroupConfig.Name) } else { libcontainerCgroupConfig.Path = cgroupConfig.Name.ToCgroupfs() } </del> <ins> libcontainerCgroupConfig.Path = cgroupConfig.Name.ToCgroupfs() </ins> } manager, err := m.adapter.newManager(libcontainerCgroupConfig, cgroupPaths)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is created on a cgroup2 host What you expected to happen: Such directory should not be created How to reproduce it (as minimally and precisely as possible): Boot Ubuntu 20.04 with Install moby/moby with containerd/containerd and opencontainers/runc Install kubernetes-sigs/kind . This version uses as the base image. Run with Run Run to drop into the shell of the node The following directories are unexpectedly created. (No process seems to belong to these groups actually) Environment: (See above)
cc
/sig node


Question: // matchStorageCapacity is a matchPredicate used to sort and find volumes func matchStorageCapacity(pvA, pvB *api.PersistentVolume) bool { <del> // skip already claimed volumes if pvA.Spec.ClaimRef != nil { return false } </del> aQty := pvA.Spec.Capacity[api.ResourceStorage] bQty := pvB.Spec.Capacity[api.ResourceStorage] aSize := aQty.Value() bSize := bQty.Value() return aSize <= bSize } <del> // filterBoundVolumes is a matchPredicate that filters bound volumes before comparing storage capacity func filterBoundVolumes(compareThis, toThis *api.PersistentVolume) bool { if compareThis.Spec.ClaimRef != nil || toThis.Spec.ClaimRef != nil { return false } return matchStorageCapacity(compareThis, toThis) } </del>

E0513 22:55:45. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 266 E0513 22:55:45. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 266 E0513 22:55:48. 957 ] watch of api.Secret ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:55:48. 957 ] watch of api.Namespace ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:55:48. 957 ] watch of api.Secret ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested W0513 22:55:49. 957 ] Resetting endpoints for master service "kubernetes" to &{{ } {kubernetes default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 6443 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50.

957 ] Resetting endpoints for master service "kubernetes" to &{{ } {kubernetes default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 6443 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} E0513 22:55:50. 957 ] unable to sync kubernetes service: endpoints "kubernetes" cannot be updated: the resource was updated to 279 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 W0513 22:55:51. 957 ] Portal net unspecified.

Defaulting to 10.0.0.0/24. I0513 22:55:51. 957 ] Will report 10.0.0.22 as public IP address. E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:01. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:08. 957 ] watch of api.Secret ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:56:08. 957 ] watch of api.Namespace ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:56:08. 957 ] watch of api.Secret ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested FAIL FAIL 111.315s
Do we know which test is failing? The last messages in the log are from , but I don't actually see a test failure message printed, and I can't recreate any failures.

The "Unable to authenticate the request due to an error: Token has been invalidated" messages are expected since the test exercises attempting to use invalid tokens () Also, the "Stop requested" messages indicate that particular test completed and invoked the deferred stopFunc, which shuts down the controllers and closes the test server
Here is the failing test: --- FAIL: TestPersistentVolumeClaimBinder (3.45s) Expected claim to be bound: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:claim02 GenerateName: Namespace:default SelfLink:/api/v1beta1/persistentVolumeClaims/claim02?namespace=default UID:-f9c3-11e4-96ec- ResourceVersion:172 CreationTimestamp:2015-05-13 22:54:47 +0000 UTC DeletionTimestamp:<nilLabels:map[] Annotations:map[]} Spec:{AccessModes:[ReadOnlyMany ReadWriteOnce ReadWriteMany] Resources:{Limits:map[] Requests:map[storage:{Amount: Format:BinarySI}]} VolumeName:} Status:{Phase:Pending AccessModes:[] Capacity:map[]}}
you aware of the issue with TestPersistentVolumeClaimBinder?
I ran into above issue a lot.
Sorry, hadn't seen this sooner. Lost in the flurry of email traffic. I am looking into it now.
I cannot reproduce on F20, Go 1.3.1 I'll try 1.4 next. Any other info would be appreciated.
Here is the output of docker version from my desktop. Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.4.2 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.4.2 Git commit (server): OS/Arch (server): linux/amd64 I started to run into this issue two days ago, and just ran into it half hour ago right after I sync-ed to HEAD again. Any more information you need here?
Thanks :)
I cannot reproduce this on F20, Go 1.4.2 Any runtime settings I might be missing? I see hack/test- is set to "-race" .


Question: <ins> // +build !providerless </ins> /* Copyright 2018 The Kubernetes Authors.

make verify WHAT=typecheck-providerless outputs /cc /milestone v1.19
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
see example successful run:
For reference -
fix in
screaming


Question: // implementing this brute force approach instead of fancy channel notification to avoid test specific code in prod. // wait for config to be observed <del> verifyIfKMSTransformersSwapped(t, wantPrefixForSecrets, test) </del> <ins> verifyIfKMSTransformersSwapped(t, wantPrefixForSecrets, "", test) </ins> // run storage migration // get secrets

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: workingSetStats[host][0], workingSetStats[host][len(workingSetStats[host])/2], workingSetStats[host][len(workingSetStats[host])-1]) } framework.Logf("Node-Problem-Detector CPU and Memory Stats:nt%snt%snt%s", cpuStatsMsg, rssStatsMsg, workingSetStatsMsg) <ins> ginkgo.By("Check node-problem-detector can post conditions and events to API server") for _, node := range nodes { ginkgo.By(fmt.Sprintf("Check node-problem-detector posted KernelDeadlock condition on node %q", node.Name)) gomega.Eventually(ctx, func() error { return verifyNodeCondition(ctx, f, "KernelDeadlock", v1.ConditionTrue, "DockerHung", node.Name) }, pollTimeout, pollInterval).Should(gomega.Succeed()) ginkgo.By(fmt.Sprintf("Check node-problem-detector posted DockerHung event on node %q", node.Name)) eventListOptions := metav1.ListOptions{FieldSelector: fields.Set{"involvedObject.kind": "Node"}.AsSelector().String()} gomega.Eventually(ctx, func(ctx context.Context) error { return verifyEvents(ctx, f, eventListOptions, 1, "DockerHung", node.Name) }, pollTimeout, pollInterval).Should(gomega.Succeed()) if checkForKubeletStart { // Node problem detector reports kubelet start events automatically starting from NPD v0.7.0+. // Since Kubelet may be restarted for a few times after node is booted. We just check the event // is detected, but do not check how many times Kubelet is started. // // Some test suites run for hours and KubeletStart event will already be cleaned up ginkgo.By(fmt.Sprintf("Check node-problem-detector posted KubeletStart event on node %q", node.Name)) gomega.Eventually(ctx, func(ctx context.Context) error { return verifyEventExists(ctx, f, eventListOptions, "KubeletStart", node.Name) }, pollTimeout, pollInterval).Should(gomega.Succeed()) } else { ginkgo.By("KubeletStart event will NOT be checked") } } </ins> }) })

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
needs containerd 1.7.14 in the COS image - specifically
Thanks, I missed that. Checking the logs from i noticed that container is destroyed ! I see also from kernel logs that docker is blocked for 120second ( around the same time the container is destroyed ) Could it be that Docker Hung should be moved to be last in the test to avoid deleting a container?
/assign
/cc
Looking at the in job name initially I thought this job runs alpha features, so they might have forgotten to disable feature. But it seems this has already taken care -
/triage accepted


Question: "testing" "code.google.com/p/go.net/websocket" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/runtime" "github.com/GoogleCloudPlatform/kubernetes/pkg/watch" )

is always using the cross namespace context. This needs to be fixed to support scoped watches.


Question: // We want also the "mynode" node since all the testing pods live there nodeName = nodeNamePrefix } <del> attachVolumeToNode("lostVolumeName", nodeName) </del> <ins> attachVolumeToNode(nodes, "lostVolumeName", nodeName) </ins> } fakeClient.AddReactor("update", "nodes", func(action core.Action) (handled bool, ret runtime.Object, err error) { updateAction := action.(core.UpdateAction)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --
/sig storage /triage accepted
Pr continue failed beacuse this data race, and i did an fix in ,it's ok now.
/assign
another instance
another instance: (PR: )
flaked again since was merged
fixed in
/reopen Can you backport to 1.21?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
opened
Hit this bug also while working on a fix for 1.19 - so maybe we should open another issue/pr for 1.20 and 1.19 ? At least 1.19 will be still around for quite a while
I think that would probably be approved. if anyone wants to take a stab at it
Thanks I'll check over weekend if i can provide this fix also to 1.20 and 1.19
Opened for 1.19 Opened for 1.20
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: --storage-media-type="${KUBE_TEST_API_STORAGE_TYPE-}"  --cert-dir="${TMPDIR:-/tmp/}"  --service-cluster-ip-range="10.0.0.0/24"  <ins> --client-ca-file=hack/testdata/ca.crt  </ins> --token-auth-file=hack/testdata/auth-tokens.csv 1>&2 & export APISERVER_PID=$!

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With the exec credentials plugin configured in my kubeconf, and a stale/expired token present in the same file issuing e.g. triggers the authentication flow as handled by the plugin - once the flow is returned to kubectl (by printing the ExecCredential object) the new token is stored to the kubeconfig but not immediately used to resolve the request, leading to: Immediately issuing again works as kubectl now seem to use the credentials (token) stored following the first request. What you expected to happen: That kubectl would read the new token as fetched by the exec credential plugin and use it already for the first command issued. How to reproduce it (as minimally and precisely as possible): the kubernetes API server with the OIDC authenticator enabled. kubectl to use an exec credential plugin printing the token received out of band. that the token was indeed written to the kubeconf file. that the first kubectl command failed. that the next identical command succeeds. Anything else we need to know?: This works as expected when there is no token previously stored in the kubeconf - the behavior is only triggered when there is a token stored but it expired, or for whatever other reason not accepted by the API server. Environment: Kubernetes version (use ): 1.14, also tried with 1.17 Cloud provider or hardware configuration: OS (e.g: ): Mac OS for kubectl client Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig auth /sig cli
could you provide more details around how is configured. For example, it is unclear to me if you are using the built in oidc plugin or a custom exec plugin. I think you are using a custom exec plugin but its also trying to write tokens to the file (which it should not do)? /triage needs-information
Thanks for looking into it . You are indeed correct in your assumption that I'm using a which stores the token in the after retrieval. I'd love to learn why this should not be done, but the problem remains also without persisting the token after retrieval.

Basically the steps to reproduce could be reduced to these: containing an expired token. same configuring an exec plugin which retrieves a valid token out of band and echoes it. Expected behavior: The valid token retrieved is used for the subsequent authentication. Possibly expected behavior: The exec plugin flow not started (or exited with a warning) due to an expired token being present in the . Current behavior seems to acknowledge the invalid token and hence tries to retrieve a new, valid one - which it then discards as soon as it arrives.
Seems we can reduce it even further by removing the token altogether from the file and instead provide it directly. So the only step needed to reproduce is essentially this: configuring an exec plugin which retrieves a valid token out of band and echoes it as part of an . Now any will trigger the exec plugin to retrieve a valid token, but will still not use it for subsequent authentication.
This one is caused by config merging with clientcmd overrides: and exec.Auth not being in this switch:
Interesting, thanks Will add it to the switch along with some tests and if everything works out well I'll submit a PR next week.
Having spent some time on this in the debugger, it seems somewhat more intricate than I first thought. Turns out the exec plugin is executed whether the presented is invalid or valid - the exec action is actually called before the first call is even dispatched. This due to the exec credential plugin possibly being needed to setup the client certificate requested for the transport. Once the exec plugin's is called it assign the authenticators function to : ..which is then called when setting up the transport, thus triggering the exec plugin regardless of a token being present or not: Since the exec plugin has no way of knowing whether a token or a client certificate will be returned by the exec action, this kind of makes sense. It however defeats letting any present to override the exec plugin, which clearly seems to be the intent: Possible options here that I can think of: (and fix) behavior of letting override the exec plugin. This looks doable by adding a check for token auth early in the 's method body: But might be there are consequences to this I'm not aware of. the behavior of letting override the exec plugin being called.

third, more complex option, would be to keep the token override but dispatch to the exec credential plugin should the response from the first call with the token come back with a 401. This was what I originally believed was the intent as my first tests always included an invalid token. While this behavior seems nice I'm more and more thinking it may violate the principle of least surprise. Either you present a token and use that for auth or you let the exec plugin fetch the credentials for you. What do you guys think?
As a colleague just made me aware of, certain tools like kubetail seem to have kubectl call the exec plugin so many times that eventually the threshold of 1000 is reached and warn logs (exec:201) start spewing - even though no certificate (and hence no rotation) should be involved. This has me more convinced that the first option is the right one, skipping the client TLS ceremonies should a token be present for authentication.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } cleanupPods(cs, t, []*v1.Pod{test.pod}) } <ins> // Close all heartbeat channels. for i := 0; i < nodeCount; i++ { close(heartbeatChans[i]) } </ins> cleanupNodes(cs, t) waitForSchedulerCacheCleanup(context.scheduler, t) })

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: TaintBasedEvictions Since when has it been failing: probably since it was in this PR? Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs -- TaintBasedEvictions probably since it was in this PR? Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17 </details
cc.
Looking into this. We moved it to integration because it was flaking in e2e too
/assign damemi
Not sure yet, but I do see this in the logs of the failing tests: which, if I understand this, would remove all the taints and mark nodes as reachable (which would explain why the test is failing waiting for node taints: ). But I need to figure out why all the nodes are becoming unavailable
is that a full fix for this flake or just partial? I still saw it failed yesterday so just checking... :)
I think I also run into this on 13th Oct: Full logs:
Now gets merged, which might fix the flake. Let's keep an eye on this.
reported it still flaked: Can you continue looking into this?
I'll keep trying, from my previous debugging it seems that in some cases the test cluster is entering full disruption mode and removing all of the taints from the nodes. I know we specifically made changes to the test (using multiple nodes) to ensure it wouldn't become fully disrupted, do you have any idea what could still be causing that?
There is some flag controlling how the dispution thing is managed internally, maybe you can check how a node lifecycle manager is constructed.

The test still flakes after we merge , we still need to investigate the cause ,
/assign
Another flake:
These flakes are consistently (always) timeouts waiting on node taints:
I'm working on this. Will send out a PR soon.
/reopen We see another flake since got merged: However, don't panic :) The original "Entering master disruption mode" cause has been fixed - you won't see that message in the above log. This time it's a smaller issue. I will explain it in another PR.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen The job is still flaky
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close But for another test :)
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ret=0 run-checks "${KUBE_ROOT}/hack/verify-*.sh" bash <del> run-checks "${KUBE_ROOT}/hack/verify-*.py" python </del> <ins> run-checks "${KUBE_ROOT}/hack/verify-*.py" python3 </ins> missing-target-checks if [[ ${ret} -eq 1 ]]; then

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: pull-kubernetes-verify Which test(s) are failing: publishing-bot Since when has it been failing: 07-31 12:23 PDT Testgrid link: Reason for failure: missing pyyaml module Anything else we need to know: Running case: verify.publishing-bot +++ working dir: +++ command: python "hack/make-rules/../../hack/verify-publishing-" Please install missing pyyaml module and re-run hack/make-rules/../../hack/verify-publishing- +++ exit code: 1 +++ error: 1
/sig testing
due to python3 bump in test-infra kubekins; being reverted in
Resolved by test-infra revert


Question: // allocate hugepages if *is2MiHugepagesSupported { <del> err := configureHugePages(hugepagesSize2M, 256) framework.ExpectNoError(err) </del> <ins> hugepagesCount := 256 ginkgo.By("Configuring hugepages") gomega.Eventually(func() error { if err := configureHugePages(hugepagesSize2M, hugepagesCount); err != nil { return err } return nil }, 30*time.Second, framework.Poll).Should(gomega.BeNil()) ginkgo.By("restarting kubelet to pick up pre-allocated hugepages") // stop the kubelet and wait until the server will restart it automatically stopKubelet() // wait until the kubelet health check will fail gomega.Eventually(func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, time.Minute, time.Second).Should(gomega.BeFalse()) // wait until the kubelet health check will pass gomega.Eventually(func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, 2*time.Minute, 10*time.Second).Should(gomega.BeTrue()) ginkgo.By("Waiting for hugepages resource to become available on the local node") gomega.Eventually(func() error { node, err := f.ClientSet.CoreV1().Nodes().Get(context.TODO(), framework.TestContext.NodeName, metav1.GetOptions{}) if err != nil { return err } capacity, ok := node.Status.Capacity[v1.ResourceName(hugepagesResourceName2Mi)] if !ok { return fmt.Errorf("the node does not have the resource %s", hugepagesResourceName2Mi) } size, succeed := capacity.AsInt64() if !succeed { return fmt.Errorf("failed to convert quantity to int64") } // 512 Mb, the expected size in bytes expectedSize := int64(hugepagesCount * hugepagesSize2M * 1024) if size != expectedSize { return fmt.Errorf("the actual size %d is different from the expected one %d", size, expectedSize) } return nil }, time.Minute, framework.Poll).Should(gomega.BeNil()) </ins> } // get the old kubelet config

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --e2e-memory-manager Different tests under the memory manager lane The node does not have configured hugepages
/sig node
/assign
/triage accepted


Question: framework.ExpectEqual(rcStatus.Status.ReadyReplicas, int32(0), "ReplicationControllerStatus's readyReplicas does not equal 0") ginkgo.By("waiting for RC to be modified") eventFound = false <del> ctx, cancel = context.WithTimeout(context.Background(), 30*time.Second) </del> <ins> ctx, cancel = context.WithTimeout(context.Background(), 60*time.Second) </ins> defer cancel() _, err = watchUntilWithoutRetry(ctx, retryWatcher, func(watchEvent watch.Event) (bool, error) { if watchEvent.Type != watch.Modified {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: - Triage link for more jobs that are affected: Which test(s) are failing: Since when has it been failing: Since October 23 Testgrid link: Reason for failure: Anything else we need to know: Example spyglass link: /sig apps /cc /priority important-soon
/cc missed this in the main body
/assign /triage accepted
/remove-priority important-soon /priority critical-urgent top flake for pull-kubernetes-conformance-kind-ipv6-parallel, 2nd top flake for per /area conformance this is a conformance test /remove-kind failing-test /kind flake this is not failing continuously, but is instead flaking
/assign
I increased the timeout while waiting for available Replicas.
Test grid green since Merge 5 Nov 2020 /cc


Question: deps = [ "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/uuid:go_default_library", </ins> "//staging/src/k8s.io/csi-translation-lib/plugins:go_default_library", ], )

Minimal Repro: Create 2 different disks Create 2 Pods each using one of the disks Start both pods on the same node You will notice one of the disks gets ed to a path: , then both disks are ed from to a target path: which is similar but at least uses the pod . Both of these volumes are being treated as the same volume in the Kubelet even though they are definitely not the same volume. Note: it may not be "treated" as the same volume - it may just be that since the paths are the same the following volumes see that the path is already a mountpoint and skip the step. This is the more likely scenario based on observations. The value is the "name" of the inline-volume. These can obviously be the same across different pods - we should actually be placing it in a more unique directory (based off the translated maybe?). /cc
/sig storage
The inline volume example I'm using (ignore the fact that is a complete misnomer): I've confirmed that changing the "name" makes inline volumes work well since there is no longer a collision on the paths
/assign
Alright, problem is sort of twofold: The referred to above for inline-volumes is set here: As we can see from the comment: it is set to the because the in-tree CSI Plugin makes a lot of naming assumptions based on the which for PVC/PVs is the PV name (unique in a cluster). Primarily, during unmount the CSI Plugin tries to from a "well known ": which happens to be some path based on the which for the inline volume is just the ( above). So we run into the problem that if we use the directory is not sufficiently unique for different disks. And if we use something else like then the unmount pathway is unable to find the . I think using is a no-go. So we need to use something sufficiently unique (like ) for the ( above) and fix up some of the other codepaths to either use different "well-known paths" to write to, or to fix up path generation code so they give us the correct locations. I'm also going to check to see whether 's refactor fixes this issue (because it might since it translates things earlier)
OK. Really good news.

your refactor in its current state actually fixes this issue if we set the to something unique here: Looks like the early translation is also making the pathway from the "correct" dir for inline migration. So I will create a PR that fixes this issue that should merge after


Question: existingNamespace.Status.Phase = api.NamespaceTerminating } <del> // Remove orphan finalizer if options.OrphanDependents = false. if options.OrphanDependents != nil && *options.OrphanDependents == false { // remove Orphan finalizer. </del> <ins> // the current finalizers which are on namespace currentFinalizers := map[string]bool{} for _, f := range existingNamespace.Finalizers { currentFinalizers[f] = true } // the finalizers we should ensure on namespace shouldHaveFinalizers := map[string]bool{ metav1.FinalizerOrphanDependents: shouldHaveOrphanFinalizer(options, currentFinalizers[metav1.FinalizerOrphanDependents]), metav1.FinalizerDeleteDependents: shouldHaveDeleteDependentsFinalizer(options, currentFinalizers[metav1.FinalizerDeleteDependents]), } // determine whether there are changes changeNeeded := false for finalizer, shouldHave := range shouldHaveFinalizers { changeNeeded = currentFinalizers[finalizer] != shouldHave || changeNeeded if shouldHave { currentFinalizers[finalizer] = true } else { delete(currentFinalizers, finalizer) } } // make the changes if needed if changeNeeded { </ins> newFinalizers := []string{} <del> for i := range existingNamespace.ObjectMeta.Finalizers { finalizer := existingNamespace.ObjectMeta.Finalizers[i] if string(finalizer) != metav1.FinalizerOrphanDependents { newFinalizers = append(newFinalizers, finalizer) } </del> <ins> for f := range currentFinalizers { newFinalizers = append(newFinalizers, f) </ins> } <del> existingNamespace.ObjectMeta.Finalizers = newFinalizers </del> <ins> existingNamespace.Finalizers = newFinalizers </ins> } return existingNamespace, nil }),

What happened: I created a namespace with finalizer, and then tried to delete it: The namespace can be deleted, and the finalizer is removed by garbage collection controller: What you expected to happen: Since kubectl species , it is not an orphaning operation, the finalizer should be removed in namespace store. How to reproduce it (as minimally and precisely as possible): Same as what I observed: a namespace. finalizer. . Anything else we need to know?: Environment: Kubernetes version (use ): HEAD
/sig api-machinery
cc
Discussion on slack:


Question: # Configure kubelet to run as a windows service. flags+=" --windows-service=true" <del> # TODO(mtaufen): Configure logging for kubelet running as a service. I haven't # been able to figure out how to direct stdout/stderr into log files when # configuring it to run via sc.exe, so we just manually override logging # config here. flags+=" --log-file=${WINDOWS_LOGS_DIR}kubelet.log" # klog sets this to true internally, so need to override to false so we # actually log to the file flags+=" --logtostderr=false" </del> # Configure the file path for host dns configuration flags+=" --resolv-conf=${WINDOWS_CNI_DIR}hostdns.conf"

/kind feature Hey, in K8s 1.23 we are planning to deprecate multiple klog flags. Deprecated flags: , , , , , , , , , , , One of the alpha milestones is requires us to remove usage of deprecated flags within K8s repo. Examples found for windows [ ] [ ] /sig windows /wg structured-logging
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I've started looking into this. Part of the problem is that go-runner is not part of kubernetes-node-windows- I suspect that is where the scripts get and kube- from.


Question: assert.Equal(t, expectedConfig, containerConfig, "generate container config for kubelet runtime v1.") runAsUser := types.UnixUserID(0) <del> RunAsNonRoot := false </del> <ins> runAsNonRootTrue := true </ins> podWithContainerSecurityContext := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID:       "12345678",

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubelet securitycontext Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.7.0-beta/1.8.0 (HEAD) Environment: Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): gci Kernel (e.g. ): Install tools: Others: docker runtime (rkt seems to have a different code path, but I haven't checked it) What happened: containers could not run as root when "RunAsNonRoot=false" What you expected to happen: containers could run as root How to reproduce it (as minimally and precisely as possible): add a securitycontext to a container, don't set privileged, do set RunAsNonRoot, use an image that runs as root. The PodSecurityPolicy admission controller will always set a securitycontext if it's not specified, and cause this to be hit more often. Anything else we need to know: The bug is in the checks here: I have a fix in but that going in depends on an extension being granted. We may want to fix the bug regardless.
/sig node
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
I'll send a fix shortly.


Question: - user: root - name: /opt/cni - makedirs: True <del> - source: https://storage.googleapis.com/kubernetes-release/network-plugins/cni-9d5e6e60e79491207834ae8439e80c943db65a69.tar.gz </del> <ins> - source: https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz </ins> - tar_options: v <del> - source_hash: md5=1ac2f8c72244f0188b8ba3d6052f9fc1 </del> <ins> - source_hash: md5=dd11b04dcb0b9aca00b5287ba18dcfaa </ins> - archive_format: tar - if_missing: /opt/cni/bin

I have a container that got the broadcast address of podCIDR subnet. My CNI conf is: And the container in question has which is the broadcast address for the subnet. So it can't figure out the MAC of cbr0 because all frames are getting sent to (or at least, that's what seems to indicate). Eg: tcpdump alonside a wget on vs a similar normal container
Looks like a bug in ipam. Will try to put together a fix.
I stress tested your fix overnight with the networking suite I'm writing (the one that found the bug), didn't hit the failure in 100 runs, we should get it in asap
Great. It would be better to pull from upstream once the fix is merged. Otherwise, we have to fork it like we did in 1.3.
Marking this p0 flake so it shows up in release queries
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: wantStatus *framework.Status }{ { <del> name:       "A pod having no tolerations can't be scheduled onto a node with nonempty taints", pod:        podWithTolerations("pod1", []v1.Toleration{}), node:       nodeWithTaints("nodeA", []v1.Taint{{Key: "dedicated", Value: "user1", Effect: "NoSchedule"}}), wantStatus: unschedulable, </del> <ins> name: "A pod having no tolerations can't be scheduled onto a node with nonempty taints", pod:  podWithTolerations("pod1", []v1.Toleration{}), node: nodeWithTaints("nodeA", []v1.Taint{{Key: "dedicated", Value: "user1", Effect: "NoSchedule"}}), wantStatus: framework.NewStatus(framework.UnschedulableAndUnresolvable, "node(s) had taint {dedicated: user1}, that the pod didn't tolerate"), </ins> }, { name: "A pod which can be scheduled on a dedicated node assigned to user1 with effect NoSchedule",

When trying to see why pods don't start using either kubectl get events or kubectl describe pod , the base message is not detailed enough. Warning FailedScheduling 5s (x4 over 65s) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. What would you like to be : Details of the taint and details of what it didn't tolerate Why is this needed: To get much much better debug information by default when trying to understand why pods can't launch.
/sig cli
/sig scheduling /priority important-longterm
/cc
/remove-sig cli
Those labels are not set on the issue: <detailsIn response to : /remove-sig cli </details
/good-first-issue
/assign starizard
Can I get an idea of what this would involve? I might be able to put some work on it.
I assume is taking this already. The error message is produced in We probably also require changes in to get the name of the missing toleration.
/remove-help


Question: Convert_versioned_Event_to_versioned_InternalEvent, ) <ins> // Register Unversioned types under their own special group scheme.AddUnversionedTypes(Unversioned, &Status{}, &APIVersions{}, &APIGroupList{}, &APIGroup{}, &APIResourceList{}, ) </ins> // register manually. This usually goes through the SchemeBuilder, which we cannot use here. scheme.AddGeneratedDeepCopyFuncs(GetGeneratedDeepCopyFuncs()...) AddConversionFuncs(scheme)

We have moved to non-global schema in a number of places in 1.6 and 1.7. It is easy to forget registering the unversioned types like , leading to encoding errors in error cases. The registration of the unversioned types should be automatic as for the types, i.e. in .
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
do we need to close out this issue? (since referenced PR has merged)
Fixed by


Question: extraArgsSeparaterEnvKey = "E2E_EXTRA_ARGS_SEP" defaultSkip         = "" <del> defaultFocus        = "[Conformance]" </del> <ins> defaultFocus        = "[Conformance]" </ins> defaultProvider     = "local" defaultParallel     = "1" defaultResultsDir   = "/tmp/results"

What happened: When using sonobuoy to run some community tests, I realized that ginkgo was not invoked properly. More precisely, when invoking sonobuoy like this: I saw the following in the sonobuoy logs: Because I am running the tests in parallel, and sonobuoy does not set , the code defaults to (see ) for , and is set to . This is not the intended regex (should be ) and all the tests are skipped (because all test names have an ). What you expected to happen: should be set to so that only the tests with the "tag" are skipped. How to reproduce it (as minimally and precisely as possible): Use the sonobuoy command provided above. Environment:
I opened a PR for this
/sig testing


Question: package master import ( <ins> "bytes" </ins> "encoding/json" "fmt" "net/http"

What happened: Created a CRD with a group/version/kind that defines a CustomResourceDefinition type overlapping with the in-tree kubernetes type (WARNING: do not create this CRD on a cluster you care about, see ) Note that two definitions for the group/version/kind appear in openapi: What you expected to happen: published openapi only includes the in-tree type, which has precedence /cc /area custom-resources /priority important-soon /sig api-machinery
/cc
haven't looked into the code yet. I'd expect the naming controller / apiextensions server to prevent conflicting CR GVK, but they are unaware of builtin API groups
CRD handling is isolated to the apiextensions server... it does not have information about all other built-in and aggregated APIs
mentioned something he was working on to make the aggregation layer honor APIService priorities (so that later definitions would not be included if something had already defined a GVK) edit: possibly
a naive idea is to reserve k8s groups in a hard code way that sounds good the CRD in defines a but the kubectl results are about . Did I miss something?
oops... fixed. I started with an API type under but the issue only surfaces for specific APIs (due to the generated definition name colliding for some types)
that wouldn't address duplicate GVKs from aggregated APIs
Three kinds of openapi paths/definitions that we want to prevent CRDs from overwriting: CRD API the spec aggregation in apiextensions-apiserver allows overwriting. I will send a fix native APIs well protected. The spec aggregation in kube-aggregator: 1) ignores duplicated paths and renames duplicated definitions; 2) treats kube-apiserver with than apiextensions-apiserver APIs the spec aggregation in kube-aggregator allows overwriting. The spec aggregation in kube-aggregator treats aggregated APIs with than apiextensions-apiserver. I'm thinking about giving apiextensions-apiserver the lowest priority, but then the problem is aggregated APIs can potentially overwrite the CRD API Note that none of the above prevents CRDs from adding new subpaths to existing APIs. E.g. a CRD can add a path to the CRD API, or to a aggregated API if it didn't have one


Question: glog.V(2).Infof("Using root directory: %v", s.RootDirectory) // TODO(vmarmol): Do this through container config. <del> oomAdjuster := oom.NewOOMAdjuster() </del> <ins> oomAdjuster := kcfg.OOMAdjuster </ins> if err := oomAdjuster.ApplyOOMScoreAdj(0, s.OOMScoreAdj); err != nil { glog.Warning(err) }

The type has a field called to indicate that a container must run as non-root user. The description for this field includes the sentence , which would mean that setting the field to or to (unset) have the same effect: However, the PodSecurityPolicy provider implementation handles the (unset) case differently, and thus does differentiate between unset or values: The API type documentation should be updated to clarify this behaviour and reflect the actual PodSecurityPolicy implementation. /sig node
Thanks for reporting this issue. I think that would be best able to assist with this. /sig auth /remove-sig node
The pod documentation is correct. If the field is not set or is false, the enforcement is not performed by the kubelet. However, podsecuritypolicy with a runasuser strategy of mustrunasnonroot will force a pod that has not specified a runAsNonRoot setting or a runAsUser uid to runAsNonRoot: true The documentation on the MustRunAsNonRoot strategy of podsecuritypolicy should clarify the mutation behavior of the PSP /remove-kind bug /kind documentation
Can I take this issue?
^ this is something that can be changed.
Here's a proposed change to the documentation in : Is this sufficient to solve this issue? If so, should I submit a PR there?
I'd change that to to be a little more complete.
/assign See PR for docs at
GitHub didn't allow me to assign the following users: josiahbjorgaard. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign See PR for docs at Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
~This would have saved me some battling today! I couldn't work out why my pod wasn't selecting my run-as-root pod security policy and ignoring my restricted pod security policy - I didn't realise I had to explicitly set to get the correct PSP to match (I thought the restrictive PSP would reject the pod and then the run-as-root PSP would match, not that the restrictive PSP would mutate the PodSpec)~ It would have had I read the actual docs better sigh.


Question: proxier.serviceNoLocalEndpointsInternal = sets.New[string]() proxier.serviceNoLocalEndpointsExternal = sets.New[string]() <ins> proxier.lbNoNodeAccessIPPortProtocolEntries = make([]*utilipset.Entry, 0) </ins> // Begin install iptables // Reset all buffers used later.

When using loadBalancerSourceRanges to allow access only from CIDRs outside the cluster segment, access from the K8s nodes (or pods with hostNetwork:true) should be rejected, but for proxy-mode=ipvs it works. /sig network /area kube-proxy /area ipvs Access from K8s nodes should be rejected in the same way as for proxy-mode=iptables Define a service with loadBalancerSourceRanges that allows access from a cidr outside the cluster, e.g. The K8s nodes have addresses on 192.168.1.0/24 and fd00::c0a8:100/120. Make sure a loadBalancerIP is assigned (in this example 10.0.0.0/32) and try to access it from a K8s node: Same for both ipv4 and ipv6. This is related to since a fix requires that the passed to the is correct. Please see v1.27.4 and master None Linux 6.4.4 None N/A N/A
wanted to work on this
Great! But please coordinate with I hope that one get accepted, and then ~the s (both families) passed to the proxiers should be usable.~ No, they are not, please see
Discussed this one in the SIG Network meeting today: we agree this is something we'd like to implement, however we weren't sure exactly how to do it with ipvs during the meeting. Whoever takes this on needs to come up with and communicate a clear plan of how it will be accomplished. /triage accepted
My comment is not feasible. For proxy-mode=ipvs the src and dst is both loadBalancerIP. So the nodeIP will not help :disappointed:
I did a small experimented with kind + metallb. I created a loadbalancer service backed by . loadBalancerIP: 172.18.255.200 podIP: 10.244.1.3 conntrack picks the src and dst both as loadBalancerIP. Tried to create an iptables rules matching source and destination with loadbalancerIP to drop packets but it didn't work. After some exploration I found and with the following rules I was able to prevent access to lbIP from k8s nodes / host network.

The solution looks hacky though, we only need this rule if source ranges are defined for the service and node ip is not covered by any of them. Will try to program this in the ipvs proxier.
Good find! :+1: I had missed the ipvs extension. I started to check nodeIP and the (flawed) option to experiment with setting a nodeIP as "src" for some routes to dodge the lbip-both-as-src-and-dst problem, and an old "vision" I have to not assign clusterIPs to at all, but have one rule to replace them all, like: It can be done, and in a cluster with 1000's of clusterIP's it would simplify


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package testing import "testing" func TestValueFuzz(t *testing.T) { type ( Y struct { I int B bool F float32 U uint } X struct { Ptr *X Y Y Map map[string]int Slice []int } ) x := X{ Ptr: &X{}, Map: map[string]int{"foo": 42}, Slice: []int{1,2,3}, } p := x.Ptr m := x.Map s := x.Slice ValueFuzz(x) if x.Ptr.Y.I == 0 { t.Errorf("x.Ptr.Y.I should have changed") } if x.Map["foo"] == 42 { t.Errorf("x.Map[foo] should have changed") } if x.Slice[0] == 1 { t.Errorf("x.Slice[0] should have changed") } if x.Ptr != p { t.Errorf("x.Ptr changed") } m["foo"] = 7 if x.Map["foo"] != m["foo"] { t.Errorf("x.Map changed") } s[0] = 7 if x.Slice[0] != s[0] { t.Errorf("x.Slice changed") } } </ins>

The apimachinery roundtripping test does not ensure that deep copies work. We need to provide a deep copy test that makes sure we don't end up with shallow copies.
I looked into gofuzz some days ago and noticed that it does not support value-only fuzzing that preserves object structure (= does not change pointer values). So we need something clever, maybe reflection based here. Any other idea?


Question: import ( "context" "fmt" <ins> "os" "path/filepath" </ins> "strconv" "time"

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node /cc
The failure should be caused by
/assign
/triage accepted /assign