Question: func TestPluginEmptyRootContext(t *testing.T) { doTestPlugin(t, pluginTestConfig{ <ins> volumeDirExists:        true, readyDirExists:         true, medium:                 v1.StorageMediumDefault, expectedSetupMounts:    0, expectedTeardownMounts: 0}) doTestPlugin(t, pluginTestConfig{ volumeDirExists:        false, readyDirExists:         false, medium:                 v1.StorageMediumDefault, expectedSetupMounts:    0, expectedTeardownMounts: 0}) doTestPlugin(t, pluginTestConfig{ volumeDirExists:        true, readyDirExists:         false, medium:                 v1.StorageMediumDefault, expectedSetupMounts:    0, expectedTeardownMounts: 0}) doTestPlugin(t, pluginTestConfig{ volumeDirExists:        false, readyDirExists:         true, </ins> medium:                 v1.StorageMediumDefault, expectedSetupMounts:    0, expectedTeardownMounts: 0})

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There is a node in my cluster whose disk is full on root partition, I cleared the disk and then rebooted the node to recover it. After the reboot, the kube-proxy on that node got stuck in "ContainerCreating" status and never changes, and all other pods can't run properly since the kubeproxy is not runing. What you expected to happen: Kubeproxy on that node can run normally after the disk clean and reboot How to reproduce it (as minimally and precisely as possible): the disk full on root partition in my env, leave roughly 100k space on root partition would make this issue more reproduciblethis issue is kind of tricky, to reproduce it, one need to populate the disk repeatedly since some disk space would be cleared because of pod evictionthe kube-proxy got deleted and created repeatedly because of pod eviction and daemon-set mechanism, it would stuck in ContainerCreating status if the disk space is not enough for creating volume contents of kube-proxy configMap volume, the first pod error event is like: MountVolume.Setup failed for volume "kube-proxy" space left on device After it got stuck, the following pod error event woud repeat over and over, kube-proxy is trying to mount configMap volume but always fail MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory the disk and reboot the reboot is neccesary to reproduce the issue, without the reboot the pod eviction would kick in to delete kube-proxy pod the kube-proxy woud stuck in ContainerCreating Status forever, restart kubelet has no effects on that, kube-proxy pod need to be mannully deleted to get recover kube-proxy pod can't mount configMap volume, the following error keeps repeating MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory Anything else we need to know?

: After digging into this issue for a quite long time, i think the reason for this issue is: The configMap volume uses emptyDir internally to create the volume directory, after the directory is created, configMap volume failed to populate it with configMap content because the disk is full, then configMap volume calls emptyDir.tearDown() to clear the temp directory. The problem is that emptyDir uses a "ready" file under /var/lib/kubelet/pods/{podId}/plugin/.. to determine whether the directory is created or not, if the ready file is present, then it wouldn't create the directory. The tearDown method only remove the directory, it doen't remove the ready file, so after the first out of disk error, the diretory creation always get skipped, hence no such file or directory error repeat over and over. Environment: Kubernetes version (use ): v.1.11.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): cent os 7 Kernel (e.g. ): Install tools: kubeadm Others:
/sig storage
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If it's any help, I just encountered this issue in kubelet v1.13.2 under Debian Stretch 9.5, after a restart. I was not out of disk space. Deleting the ready files from /var/lib/kubelet/pods/{podId}*/ helped.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen hit this just now with the same conditions as originally described, but without a reboot.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen </details
/reopen
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks ! this helped me get my cluster running after all pods were evicted and clearing space alone didn't fix the issue.


Question: return nil, false, nil } <ins> // Cannot run cloud ipam controller if cloud provider is nil (--cloud-provider not set or set to 'external') if controllerContext.Cloud == nil && controllerContext.ComponentConfig.KubeCloudShared.CIDRAllocatorType == string(ipam.CloudAllocatorType) { return nil, false, errors.New("--cidr-allocator-type is set to 'CloudAllocator' but cloud provider is not configured") } </ins> clusterCIDRs, err := validateCIDRs(controllerContext.ComponentConfig.KubeCloudShared.ClusterCIDR) if err != nil { return nil, false, err

Cloud-providers started to move out of tree its controller, but they share some functionalities, one of them is the NodeIPAM controller. There is a type that is GCE specific, the type. If this type is used, but no configuration for GCE is passed, the kcm panics. This can be observed in the following job: The question is, should this type be removed or a noop in KCM?
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
/sig cloud-provider
Is this a case of CCM handling IPAM so KCM needs to not handle it? What I mean is why is this set without a config? What was the intention? The fact that this has GCE-cloud-provider baked in is awful and should probably have been inverted (GCE provider calls a "register" method or something), but that's aside
Context: I started to set up jobs using the CCM by default, the cluster set up uses the script. The script doesn't consider IPAM, so it configures both KCM and CCM, but if set to "external" , it doesn't configure GCEPROVIDER in KCM When , the scripts set ... and CloudAllocator only runs with GCE My concern is that if this happens to me, it is possible that users that have to migrate custom cluster will face the same issue
A panic here seems appropriate, right? They specifically asked for a config that doesn't work and action needed to be taken. Am I misunderstanding?
A panic is appropriate given we never did any handling of this corner case -- I think it would be reasonable going forward to have a check in place to disable nodeipam controller if the allocator is but the cloud provider is not configured.
Opened with the check and a warning when this happens. We can also consider erroring out KCM if we see this happen so it's really clear to users that they misconfigured something


Question: } } <ins> // Set OODCondition for the node. func (kl *Kubelet) setNodeOODCondition(node *v1.Node) { currentTime := metav1.NewTime(kl.clock.Now()) var nodeOODCondition *v1.NodeCondition // Check if NodeOutOfDisk condition already exists and if it does, just pick it up for update. for i := range node.Status.Conditions { if node.Status.Conditions[i].Type == v1.NodeOutOfDisk { nodeOODCondition = &node.Status.Conditions[i] } } newOODCondition := nodeOODCondition == nil if newOODCondition || nodeOODCondition.Status == v1.ConditionUnknown { nodeOODCondition = &v1.NodeCondition{ Type:               v1.NodeOutOfDisk, Status:             v1.ConditionFalse, Reason:             "KubeletHasSufficientDisk", Message:            "kubelet has sufficient disk space available", LastTransitionTime: currentTime, } } // Update the heartbeat time irrespective of all the conditions. nodeOODCondition.LastHeartbeatTime = currentTime if newOODCondition { node.Status.Conditions = append(node.Status.Conditions, *nodeOODCondition) } } </ins> // Maintains Node.Spec.Unschedulable value from previous run of tryUpdateNodeStatus() // TODO: why is this a package var? var oldNodeUnschedulable bool

Both https://k8s- and https://k8s- started failing recently. Their 1.7 counterparts are passing. The difference between the last passing and the first failing tests is: Probably something in broke the tests. /sig scheduling /cc /kind bug
It looks like there isn't enough cpu on the master to run the cos-nvidia-installer daemonset. Did we add any addons recently?
My previous comment is a red herring. It is an issue with . Ill post a fix soon. From the scheduler logs on the master:
that's a benign error. The daemonset is not expected to run on the master. But there is no easy way to express that request in our APIs.
Yes, the pod not scheduling on the master is benign. The actual error is that we do not report node condition , so it is left as . The fix is to report that is false.
The tests are still failing. /reopen
/assign
I will be updating to track progress.
tests are green again! /close


Question: } return fmt.Sprintf("%s-%d", base, id) } <ins> func checkApiServiceListQuantity(aggrclient *aggregatorclient.Clientset, label string, quantity int) func() (bool, error) { return func() (bool, error) { var err error framework.Logf("Requesting list of APIServices to confirm quantity") list, err := aggrclient.ApiregistrationV1().APIServices().List(context.TODO(), metav1.ListOptions{LabelSelector: label}) if err != nil { return false, err } if len(list.Items) != quantity { return false, err } framework.Logf("Found %d APIService with label %q", quantity, label) return true, nil } } func checkApiServiceStatus(aggrclient *aggregatorclient.Clientset, apiServiceName string, statusConditions apiregistrationv1.APIServiceCondition) func() (bool, error) { return func() (bool, error) { framework.Logf("Get APIService %q to confirm status", apiServiceName) currentApiService, err := aggrclient.ApiregistrationV1().APIServices().Get(context.TODO(), apiServiceName, metav1.GetOptions{}) if err != nil { return false, err } for _, cond := range currentApiService.Status.Conditions { if cond.Type == statusConditions.Type && cond.Reason == statusConditions.Reason && cond.Message == statusConditions.Message { framework.Logf("APIService %q has the required status conditions", apiServiceName) return true, nil } } return false, err } } </ins>

[X] Test approval issue: [X] Test PR: 7 July 2022 [x] Two weeks soak end date: 21 Jul 2022 [x] Test promotion According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. replaceApiregistrationV1APIServiceStatuspatchApiregistrationV1APIServiceStatusreplaceApiregistrationV1APIServicedeleteApiregistrationV1CollectionAPIService Using a number of existing e2e test practices a new has been created for 4 APIService endpoints. The e2e logs for this test are listed below. This query shows the following endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: detect-master > /dev/null if [ "$KUBE_MASTER_IP" != "" ] && [ "$KUBERNETES_MASTER" == "" ]; then <del> echo "Setting master to: https://${KUBE_MASTER_IP}" </del> export KUBERNETES_MASTER=https://${KUBE_MASTER_IP} fi

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-master-scale-performance Which test(s) are failing: load Since when has it been failing: 11/26 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig scalability /priority critical-urgent /milestone v1.18
/assign


Question: 1. If this is your first time, read our contributor guidelines https://git.k8s.io/community/contributors/guide#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide 2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here: https://git.k8s.io/community/contributors/devel/release.md#issue-kind-label <del> 3. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews 4. Follow the instructions for writing a release note: https://git.k8s.io/community/contributors/guide/release-notes.md 5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests </del> <ins> 3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/testing.md 4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews 5. Follow the instructions for writing a release note: https://git.k8s.io/community/contributors/guide/release-notes.md 6. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests </ins> --> **What type of PR is this?**

I propose we add guidance on how to run tests for a PR in the PR template. I am a new contributor and took me much (much!) looking around to find testing instructions. It seems that page contains everything one needs to know about testing. I think it is very fitting to have it included in the template, as a one of the last check items. I propose the template be changed from this: this: /kind documentation
/sig docs
I agree with you.
/sig contributor-experience cc:


Question: } // Exists checks if a certificate embedded in kubeConfig file exists <del> func (rw *kubeConfigReadWriter) Exists() bool { </del> <ins> func (rw *kubeConfigReadWriter) Exists() (bool, error) { </ins> return fileExists(rw.kubeConfigFilePath) }

When the command is run with superuser privileges, it prints the requested information. Print a warning that files could not be read due to insufficient permission, or that information was missing (e.g., when the /etc/kubernetes directory is not initialized). When the directory is readable, it works as expected: where OS and kubeadm version detail was: Issue the command as a normal user when the directory /etc/kubernetes/pki exists but not readable. No response <details</details<detailsBare metal </details<details</details<detailsapt-get install kubeadm (source repo: ) </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle


Question: return nil, errs.ToAggregate() } } <del> return newEventRateLimit(configuration, realClock{}) </del> <ins> return newEventRateLimit(configuration, clock.RealClock{}) </ins> }) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --While working on , I realized that implements another rather than using the one already implemented in I'm not sure why this was re-implemented, but if we can migrate to using itself, then the code in the package can be cleaned up significantly (as part of ). I expect areas of the codebase that make use of a real clock to consistently use instead of re-implementing it NA Another (slightly) related issue: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/remove-kind bug /kind cleanup /sig auth
/cc fyi
/assign
Better yet, use the one in
/sig api-machinery
/assign /triage accepted


Question: rc.RLock() defer rc.RUnlock() <del> return rc.handlers </del> <ins> var copyHandlers = make(map[string]cache.PluginHandler) for pluginType, handler := range rc.handlers { copyHandlers[pluginType] = handler } return copyHandlers </ins> } func (rc *reconciler) reconcile() {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master Which test(s) are flaking: //pkg/kubelet/cm/devicemanager/godefaulttest:run2of_2 Testgrid link: Reason for failure: Anything else we need to know: links to appreciated Triage shows nothings (seems to not have updated data), but there are multiple failures in the job bazel-test-master. links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
/sig release
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-release Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Digging further as the step has another recent test fail ... So on the two runs reported by has 2 test failures has 1 test failure And the most recent failure in this area is a TestAdmit/match test failure but on a different test.
/sig node
/assign /triage accepted I'll take a look at them and fix.


Question: // Ensure cbr0 has no conflicting addresses; CNI's 'bridge' // plugin will bail out if the bridge has an unexpected one <del> plugin.clearBridgeAddressesExcept(cidr.IP.String()) </del> <ins> plugin.clearBridgeAddressesExcept(cidr) </ins> } }

This is at least partially responsible for - some tests can't run when dns is down. Logs from just before the crash: It got killed because Kubelet decided to kill it (readiness probe failed): It started happening quite recently - I'm trying to locate a build which caused this. Repro: run "Kubelet should not restart containers across restart" test - it caused this behavior for me multiple times today. cc - I applied the P0/v1.3 label/milestone, as I think it should be a blocker. Feel free to decrease the priority if you disagree
gce-serial 1511 seems to be the first one with this problem.
Or 1510 - hard to tell, as it was a broken run. There's no PR around that looks too suspicious though:/
I locally reverted and it seems that DNS is no longer failing. Sadly I have to run to catch a train, but maybe someone can pick it up here ?
Taking a look now. The PR should be unrelated...I am checking to see if the Federation DNS changes went in yet. Add nethealth container to prepull manifest
I search the restart counts for jobs from 1510-latest and gce-serial Job 1540 is the first one to show a non-zero kubedns restart count. If there is a commit responsible, it would be between 1539 and 1540 (June 21 7 AM -11 AM) 1539 end of run 1540 end of run Latest
The kubedns:1.3 container hasn't been updated since (Sat, 04 Jun 2016 03:29:37.847 GMT). GET
cc The kubedns container code is from June 04th. Dawn suggested that its networking issues that are causing kubedns container to fail its readiness check. I will need to add logic to print kubedns container logs during e2e tests, I am trying to search for any logs that might help, otherwise I will have to debug the next run live.
As reported, a simple kubelet restart triggers this issue - it looks like 10.0.0.1 is unreachable from the kubedns container.

The kubedns container blocks indefinitely waiting for a connection to the kubernetes service at 10.0.0.1 before setting up the healthz handlers - the readiness probe will fail until kubedns successfully connects to the API server.
- you can try reverting and running it again - it worked for me for some reason (it might be by accident though)
I think the root cause is that cbr0 doesn't have an IP address assigned after kubelet restart - all the pods running behind cbr0 are isolated from the network. Not only kube-dns, but after my kubelet restart, the other addon pods are also stuck in a crash loop.
This sounds like a blocker even more now.
Looks like a regression. I am investigating.
Found the root cause. There is a subnet ip mismatch. vs . Hence kubenet thinks they are different and clears the address. Will submit a fix soon.
i found coredns status is CrashLoopBackOff and i check the logs : i got tips from so i restart my docker , i check the logs , there is no error in logs again


Question: t.Error(err) } pointsTests := []jsonpathTest{ <del> {"exists filter", "{[?(@.z)].id}", pointsData, "i2 i5"}, {"bracket key", "{[0]['id']}", pointsData, "i1"}, </del> <ins> {"exists filter", "{[?(@.z)].id}", pointsData, "i2 i5", false}, {"bracket key", "{[0]['id']}", pointsData, "i1", false}, </ins> } testJSONPath(pointsTests, false, t) }

On , will fail to evaluate a filter when the member does not exist: For instance, if I were to execute Where not all in that namespace had the annotation, this would fail with the error message: This is confusing as it leads me to think more than one item was selected, so while it is not technically incorrect the error message is misleading. I would furthermore expect to more gracefully handle this case -- a missing here means that the filter should fail, not error.
What do you mean by ? How is it different from ?
As in, don't add that item to the list of items on which the following access is made. When I do: And there is no annotation on , I just get nothing back -- no error, just nothing. Similarly, i would have expected the filtered expression to give me the for all of the pods that have an annotation with value . That, anyway is what I understand the JSONPath syntax to mean. When you write You are saying:
cc since you authored and I'm not sure how we want to fix this.
Taking a look now
it looks like you need to do this (note that the annotations portion is wrapped by ): I wrote a unit test and it worked as expected. PTAL!
I'm pretty sure that was a typo on my end but let me check -- do you have a reproducer script?
I see an entirely different error today... weird:
Ok so you have 1 object with annotations, and 1 without. Let me look into that case.
Let me add another annotation to
No difference if both have annotations:
Ok, I'll have to test using kubectl
I'm the Origin bundled but I don't think there are any patches on JSONPath there:
I'm able to reproduce in the unit test when 1 object has annotations and the other does not, if allowMissingKeys is true for the jsonpath parser.
I see. Let me know if I can be of more help.
I think I have a fix. PR coming soon.
Possible


Question: return KeyError{obj, err} } <del> // If object is supposed to be deleted (last event is Deleted), // then we should ignore Sync events, because it would result in // recreation of this object. if actionType == Sync && f.willObjectBeDeletedLocked(id) { return nil } </del> newDeltas := append(f.items[id], Delta{actionType, obj}) newDeltas = dedupDeltas(newDeltas)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I did a review of and found out a case where a SharedIndexInformer can miss a notification. Follows an example scenario where this could happen. 1) A SharedIndexInformer is running, and its reflector is WATCHing. 2) A delete notification for an API object X with namespaced name N1 and UID U1 which was in the informer's cache is received. 3) The reflector on X; the relevant store is a DeltaFIFO and its Delete method is . 4) The DeltaFIFO's Delete method invokes , which . Notice that the store of the reflector, i.e., the DeltaFIFO, is NOT the SharedIndexInformer's store/cache: . Assume that BEFORE X's deletion is popped from the DeltaFIFO and applied to the informer's cache, the following sequence of events take place: 5) The WATCH is interrupted because of a network partition. 6) While the reflector is not watching, a new API object Y with namespaced name N1 (same as X) and UID U2 is created. 7) The partition disappears and the reflector does a relist against the API server; there's a "hole" between the last watch notification and the list results; assume Y is in the LIST results. 8) The reflector will load the LIST results in the store/DeltaFIFO with which in turn calls the DeltaFIFO's method. The DeltaFIFO's replace method attempts to add to the DeltaFIFO a delta of type for every object in the LIST results: when processing Y it will which should . But won't enqueue Y's delta, because it enqueues deltas of type only if there's not already a delta of type for the given object ID as the last delta for that object ID (see and ), to avoid recreating a deleted object during a periodic resync triggered by the SharedIndexInformer resync period. Object IDs are namespaced names, and remember that the last delta for Y's namespaced name in the DeltaFIFO is X's deletion.

The consequence is that Y's creation delta is not to the DeltaFIFO: Y is not to the SharedIndexInformer's cache and its creation notification handler does not execute. The SharedIndexInformer missed Y's creation. What you expected to happen: I expected SharedIndexInformers to get all notifications from the API server. Anything else we need to know? Missed notifications should be rare: for them to occur a delta must stay in the DeltaFIFO without being processed for long enough for a new List against the API server to be completed. Also, touches the reflector code that handles list and watches, and it might introduce changes relevant to this bug. Proposed solution: There's a straightforward fix: get rid of the check that skips enqueuing of a delta for object ID X if the last delta to the DeltaFIFO (and still in it) for X is of type ; is the code to remove. Notice that such check applies to deltas of type only. deltas are triggered only by relists and periodic resyncs. In the case of relists the check is harmful as it's the cause of this bug. In the case of periodic resyncs, that check is useless because it will either evaluate to or not execute; in fact, during a period resync, and this condition is checked before the check that would be removed, OTOH, if this condition is true and processing goes on the check that would be removed can only evaluate to (hence the delta for X is enqueued). If this solution has consensus I'd be happy to open a PR.
/assign
/sig api-machinery
cc
cc
xref ,
cc
/cc
This looks similar to failures we were seeing in kube-proxy while doing a rolling master upgrade. It was repeatable enough that someone hacked a workaround that I later spotted - this is close enough to the description that I’m suspicious that it was real /priority critical-urgent Since any missed informer event can cause all sorts of safety violations.
Note that rolling master upgrades break watches en masse, so a backlogged informer (contention / other locking / busy machine) would be more likely so see it the more caches it has.
A great first step would be to craft a test that demonstrates the issue repeatably, to help verify potential fixes resolve it.

IYO what would be the most appropriate kind of test, a unit test for the DeltaFIFO or something higher-level like integration/e2e tests?
The lower-level the test will be the better. So ideally unit test.
the lowest level that demonstrates the problematic behavior. if the issue is with the interaction between reflector and shared informer, then ideally something at the shared informer level
When the reflector calls it is expected that for each object O with ID X in a delta is to the DeltaFIFO, but that is (erroneously) not the case if the DeltaFIFO already stores a delta for ID X . So it's the DeltaFIFO which causes troubles and should be tested. But that alone does not show the user-visible problem; for a test to fail because a SharedIndexInformer missed a notification, the test would have to be at the SharedIndexInformer level.
That seems reasonable. I'd expect a DeltaFIFO test that sets up the described scenario and exercises both the Relist/Replace and Resync cases described here:
Sounds good, I'm at work.
is the testcase literally just adding this to But for some reason this hangs in the Pop(f).
Yes, the test looks very similar to what you posted, but there's a mistake: the expected result must be a single with the two that in your example are the only elements of the two 1-element Deltas that make up , respectively.
(that would be caught and fail the test immediately by changing to )
Aha, you're totally right :) Had to run off to lunch before spending more time on it.
DeltaFIFO only de-dupes deletions, but your example would do effectively extend that to compressing Delete+Sync=Sync. You used to be able to build smarter compression off DeltaFIFO (eg ) but nobody actually used the DeltaCompressor stuff so we removed it. And nobody above DeltaFIFO expects compressed events now :( In your example I'd be a bit worried about the implications of not deleting the object, since the Sync-ed one probably has a different UUID and thus technically not be the same object as the deleted one. Might not be a big issue, but this is critical path code so I'd personally rather be a bit conservative.
I agree about not compressing.

It's a little hard to reason about what this sync event should be when we know that relists can currently time-travel backwards. Edit: it looks like shared informer code will if it follows a delete event.
I was this morning in the "sig-api-machinery" channel of the "Kubernetes" Slack team. My reading of the code had me expecting that I'd get a delete followed by an add.
This issue is specifically focused on how a queued delete event (typically via directly observing a delete watch event or via absence in a relist/replace) followed by an observed add (via presence in a relist) is handled by DeltaFIFO. your question is slightly broader/different, and has multiple facets: UID changes considered when handling reflector object add/delete/update events? They are not; if an object is deleted and recreated between the reflector dropping a watch and relisting, the reflector will enqueue an Sync event with the object with the new uid (which the shared informer will turn into an Update event) delete/add events is actively compressed into an update event? I don't think so, but it's been a while since I've traipsed through reflector/fifo code, so I'm not positive (and since the Update with a new uid can occur from scenario 1, informer clients have to be prepared for uid to change on update anyway)
Listeners need to handle Sync too. And the only two in-tree users of NewDeltaFIFO are and and they look like they correctly handle Sync.
Edit: yep you're right, I was looking at slightly older code.
if this happens, I think it should happen on top of the DeltaFIFO, not in the DF. Let's keep DF simpler since it's (a) complicated enough and (b) critical path code. probably the right place for compression, if any happens. IMHO it's correct that Deletes don't get compressed, eg that Delete + Add is not compressed into Update or something like that.
I agree, though that specific example can happen naturally if it occurs in a window while the reflector isn't getting watch events.

Just to make sure I understand this properly, does this scenario involve an object in the cache with name "x" and UID "u1," then the reflector loses its watch, reestablishes it and lists all the relevant objects in the cluster, finds one named "x" with UID "u2," and since it already has an "x" in its cache, translates this discovery into a call?
exactly (both namespace and name are part of the cache key, uid is not)
does this mean we couldn't use the argument to DeltaFIFO's Replace() to work around whatever is talking about (if that problem still exists)?
clients can only use resourceVersion to check equality, not older-than/newer-than
As noted in Slack, there is now careful commentary in about what is promised. There is no promise that a delete followed by an add is delivered as two notifications; it is allowed to be delivered as one update notification.


Question: } } <del> legacyEndpointExpectedStatus := 200 if err := checkURL("http://metadata.google.internal/computeMetadata/v1/instance/attributes/disable-legacy-endpoints", h, 200, "true", ""); err == nil { // If `disable-legacy-endpoints` is set to true, queries to unconcealed legacy endpoints will return a 403. legacyEndpointExpectedStatus = 403 } for _, e := range legacySuccessEndpoints { if err := checkURL(e, h, legacyEndpointExpectedStatus, "", ""); err != nil { log.Printf("Wrong response for %v: %v", e, err) success = 1 } } </del> xForwardedForHeader := map[string][]string{ "X-Forwarded-For": {"Somebody-somewhere"}, }

Warning: The v0.1 and v1beta1 metadata server endpoints are deprecated and scheduled for shutdown on September 30, 2020. This test (which runs in most e2e jobs that run on GCE): Schedules a container that runs this: Resulting in a number of legacy metadata endpoints getting hit. To see if this test is going to start failing across all release branches on Sep 30th, we could try disabling the endpoints ahead of time via: In fact and support for expecting different results based on whether is set. My concern is the test expects 200 for some root endpoints, and 403 for disabled endpoints. However the e-mail I received as a GCP customer tells me any legacy endpoints may return 404. I'm inclined to suggest we remove the legacy endpoints from the check, bump the image version, and cherry-pick use of the new image version back to all release-branches. WDYT since you were involved with the earlier PRs /sig auth /sig cloud-provider /provider gcp /sig testing /priority important-soon (I'm tempted to critical-urgent this but it's not drop-everything yet) /milestone v1.20
/kind cleanup
Heyho I wanted to check how it is going? The code freeze is starting 12.11.2020 (about 3-4 weeks from now) and while there is still plenty of time, we want to ensure that each relevant issue has a chance to be considered. As the issue is tagged for 1.20, is it still planned for this release?
Bug Triage here, we're pretty late into code freeze and it doesn't look like there has been much activity recently. I am going to clear the milestone on this issue. /milestone clear
any update here?
Try to cleanup the legacy endpoints in
/milestone v1.21
/reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: )
Reopened this issue. <detailsIn response to : /reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: ) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /assign


Question: // CanIOptions is the start of the data required to perform the operation.  As new fields are added, add them here instead of // referencing the cmd.Flags() type CanIOptions struct { <del> AllNamespaces bool Quiet         bool NoHeaders     bool Namespace     string AuthClient    authorizationv1client.AuthorizationV1Interface </del> <ins> AllNamespaces   bool Quiet           bool NoHeaders       bool Namespace       string AuthClient      authorizationv1client.AuthorizationV1Interface DiscoveryClient discovery.DiscoveryInterface </ins> Verb           string Resource       schema.GroupVersionResource

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What you expected to happen: if returns true I expect to be able to perform the action without issue. Note in this case my service user has a rolebinding in a single namespace (and that is also what my kubecontext is set to.) if I change my default to a different namespace then the command returns false How to reproduce it (as minimally and precisely as possible): create a namespace (test) create a service account in that namespace create a role that gives get list and watch permissions to create a rolebinding from that role to the service account. Anything else we need to know?: this is special because namespaces are not scoped to a namespace. if you replace the command with then you get the exepcted result however this breaks the path of least suprise. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig auth
asks the authorization system if the given verb can be performed on the given noun at the given scope. It does not interpret whether the resource exists at that scope. We could warn if we find the specified resource and it does not exist at the specified scope (just like we warn if you ask about a resource that we can't find), but we would still interrogate the authorization system and return the results as we do today. /kind feature /remove-kind bug
/cc


Question: // always be verbose on failure if len(failedChecks) > 0 { klog.V(2).Infof("%s check failed: %sn%v", strings.Join(failedChecks, ","), name, failedVerboseLogOutput.String()) <del> http.Error(responsewriter.GetOriginal(w), fmt.Sprintf("%s%s check failed", individualCheckOutput.String(), name), http.StatusInternalServerError) </del> <ins> httplog.SetStacktracePredicate(r.Context(), func(int) bool { return false }) http.Error(w, fmt.Sprintf("%s%s check failed", individualCheckOutput.String(), name), http.StatusInternalServerError) </ins> return }

panics with error when health probes time out. The following stack trace shows this error (from an apiserver that uses : The is thrown by net/http because attempts to call after the request had timed out (handler had completed) here is the code site that triggers the panic (on master): Is there any historical context why we use the original by calling . If we use the decorated object then the panic will be prevented: uses the underlying object only when the request handler has not completed yet. No panic hard to reproduce, maybe we should add a unit test to reproduce the issue.
/sig api-machinery cc
/assign
GitHub didn't allow me to assign the following users: benluddy. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted


Question: var wg sync.WaitGroup var failMu sync.Mutex failed := false <ins> if *serial { *parallel = 1 } else if *parallel == 0 { *parallel = len(plats) } throttle := make(chan int, *parallel) </ins> for _, plat := range plats { wg.Add(1) <del> fn := func(plat string) { </del> <ins> go func(plat string) { // block until there's room for this task throttle <- 1 defer func() { // indicate this task is done <-throttle }() </ins> f := false serialFprintf(os.Stdout, "type-checking %sn", plat) errors, err := c.verify(plat)

Background: We recently introduced Guaranteed Pod QOS on jobs running in GCP project (). Introducing limits on jobs naturally started causing some failures, one of which was (). The original memory limit of caused the pod to be unscheduleable, so it was bumped down to () at which point the pod became scheduleable and the job ran successfully (). At that point, after observing metrics, I mistakenly concluded that this was much higher than needed and suggested bumping down again (). This caused the job to continue to run, but to fail on and sometimes . was recently refactored to not depend on GOPATH (). This was a fairly significant change so I looked at the metrics before and after the rewrite was merged. (June 25th) ! (June 27th) ! Memory usage nearly double for post-merge, which appears to be directly related to the rewrite given that on other branches remained relatively constant. This can be affirmed further by the fact that and are failing with the new resource limits, while older release branches are not. I can see a few options for how we may be able to remedy this issue in the short term: Revert the rewrite to get back to lower (yet still very high) memory usage Separate into a separate job to not reduce signal on the rest of Bump memory limits back up to to get this running successfully again for now I plan to look into fixing up but think it would be a good idea to take one of the directions mentioned above if this cannot be remedied quickly. /priority critical-urgent /sig testing /cc
latest run is green
hmm


Question: return nodeIP, nodeIPExists } <del> func getInstanceTypeByProviderIDOrName(ctx context.Context, instances cloudprovider.Instances, node *v1.Node) (string, error) { instanceType, err := instances.InstanceTypeByProviderID(ctx, node.Spec.ProviderID) </del> <ins> // getInstanceTypeByProviderIDOrName will attempt to get the instance type of node using its providerID // then it's name. If both attempts fail, an error is returned. func getInstanceTypeByProviderIDOrName(ctx context.Context, instances cloudprovider.Instances, providerID, nodeName string) (string, error) { instanceType, err := instances.InstanceTypeByProviderID(ctx, providerID) </ins> if err != nil { providerIDErr := err <del> instanceType, err = instances.InstanceType(ctx, types.NodeName(node.Name)) </del> <ins> instanceType, err = instances.InstanceType(ctx, types.NodeName(nodeName)) </ins> if err != nil { return "", fmt.Errorf("InstanceType: Error fetching by providerID: %v Error fetching by NodeName: %v", providerIDErr, err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: introduces a regression that func is invoked on a Node obj without , hence it can fail fetching the node addresses when the is not set. What you expected to happen: I would expect cloud-controller-manager to be able to initialize new Node. How to reproduce it (as minimally and precisely as possible): the gcp legacy-cloud-controller v1.18.1 () that the node-controller cannot initialize a new Node. Logs of the cloud-controller-manager repeated multiple times: Ensure that the Node is with the taint. Ensure that the of the Node is not set. The cloud-controller-manager is not running on the host machine and that's why it cannot call the gce matadata. But it was never a hard requirement for cloud-controller-manager to run on the host machine. Anything else we need to know?: The legacy cloud-controller-manager v1.17.x is working perfectly fine. Environment: Kubernetes version (use ):v1.18.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /kind regression
/cc
can we open a new issue re:


Question: framework.Failf("failed to get back the correct flunders list %v from the dynamic client", unstructuredList) } <ins> ginkgo.By("Read Status for v1alpha1.wardle.example.com") statusContent, err := restClient.Get(). AbsPath("/apis/apiregistration.k8s.io/v1/apiservices/v1alpha1.wardle.example.com/status"). SetHeader("Accept", "application/json").DoRaw(context.TODO()) framework.ExpectNoError(err, "No response for .../apiservices/v1alpha1.wardle.example.com/status. Error: %v", err) var jr *apiregistrationv1.APIService err = json.Unmarshal([]byte(statusContent), &jr) framework.ExpectNoError(err, "Failed to process statusContent: %v | err: %v ", string(statusContent), err) framework.ExpectEqual(jr.Status.Conditions[0].Message, "all checks passed", "The Message returned was %v", jr.Status.Conditions[0].Message) ginkgo.By("kubectl patch apiservice v1alpha1.wardle.example.com -p '{"spec":{"versionPriority": 400}}'") patchContent, err := restClient.Patch(types.MergePatchType). AbsPath("/apis/apiregistration.k8s.io/v1/apiservices/v1alpha1.wardle.example.com"). SetHeader("Accept", "application/json"). Body([]byte(`{"spec":{"versionPriority": 400}}`)).DoRaw(context.TODO()) framework.ExpectNoError(err, "Patch failed for .../apiservices/v1alpha1.wardle.example.com. Error: %v", err) err = json.Unmarshal([]byte(patchContent), &jr) framework.ExpectNoError(err, "Failed to process patchContent: %v | err: %v ", string(patchContent), err) framework.ExpectEqual(jr.Spec.VersionPriority, int32(400), "The VersionPriority returned was %d", jr.Spec.VersionPriority) ginkgo.By("List APIServices") listApiservices, err := restClient.Get(). AbsPath("/apis/apiregistration.k8s.io/v1/apiservices"). SetHeader("Accept", "application/json").DoRaw(context.TODO()) framework.ExpectNoError(err, "No response for /apis/apiregistration.k8s.io/v1/apiservices Error: %v", err) var list *apiregistrationv1.APIServiceList err = json.Unmarshal([]byte(listApiservices), &list) framework.ExpectNoError(err, "Failed to process APIServiceList: %v | err: %v ", list, err) locatedWardle := false for _, item := range list.Items { if item.Name == "v1alpha1.wardle.example.com" { framework.Logf("Found v1alpha1.wardle.example.com in APIServiceList") locatedWardle = true break } } framework.ExpectEqual(locatedWardle, true, "Unable to find v1alpha1.wardle.example.com in APIServiceList") </ins> // kubectl delete flunder test-flunder err = dynamicClient.Delete(context.TODO(), flunderName, metav1.DeleteOptions{}) validateErrorWithDebugInfo(f, err, pods, "deleting flunders(%v) using dynamic client", unstructuredList.Items)

[x] test pr : kuberenetes/kubernetes# [x] two weeks soak start date : testgrid-link [x] two weeks soak end date : [x] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. With this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Addressing the following endpoints in this document; Using the current e2e test as a template, create an APIService for Using the REST client, read the APIService status (readApiregistrationV1APIServiceStatus) and confirm the value of . Using the REST client, patch the APIService (patchApiregistrationV1APIService) before confirming that it has been updated. Using the REST client, list all APIServices in the cluster (listApiregistrationV1APIService) before confirming that can be found. Clean up APIService resources used by the test. Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. This query shows the endpoints hit within a short period of running the e2e test example endpoint | useragent ---------------------------------------|
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator2 (6 rows) Based on the initial APIsnoop query and the results listed above there are three new endpoints hit.

If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted


Question: return result, err } <del> type UnstructuredDefaulter struct{} </del> <ins> type unstructuredDefaulter struct { delegate runtime.ObjectDefaulter } </ins> <del> func (UnstructuredDefaulter) Default(in runtime.Object) {} </del> <ins> func (d unstructuredDefaulter) Default(in runtime.Object) { // Delegate for things other than Unstructured. if _, ok := in.(runtime.Unstructured); !ok { d.delegate.Default(in) } } </ins> type CRDRESTOptionsGetter struct { StorageConfig           storagebackend.Config

BUG REPORT Kubernetes version (use ): v1.6.2 Environment: Cloud provider or hardware configuration: GKE What happened: Create TPR "MyResource" Create namespace "soak-test" Create a "MyResource" "a" under "soak-test":Delete namespace "soak-test"wait namespace to be deleted MyResource "a" still exists after ns is deleted.
/area third-party-resource
/sig cli
I'd like to take a look at the problem if nobody else is working on this
Root Cause of this is the fact that Namespace on ThirdPartyResources is getting ignored (both passed as command --namespace, or the one present in yaml) ThirdPartyResources always end up in default namespace (which is causing confusion reported in bug)
How long has that bug existed? might related to your other findings. We know that bug doesn't exist in CRDs (we have tests specifically checking namespace handling), so I'd suggest converting to those as soon as possible.
Isn't CRD available in 1.7+ ?
Yes. If the bug has existed for multiple releases (not a regression), I don't think this is a blocker for 1.7. It could still be patched, just not a blocker for the current release.
Migration to CRD still is not automated and will still take some time. Have you considered that?
Yes, but if the behavior is not a regression, then this does not block the release. The fact that we have a clear path forward to a working resource is gravy.
This bug existed in 1.4 (), got fixed in 1.5, but appears again in 1.6 .
Ok. Not a blocker for 1.7, but if the patch is reasonable as a fix we can merge it into the service stream. For 1.7, I'd still recommend moving to CRD as quickly as possible.
I was able to reproduce this with CRD on master HEAD. What level of tests are there? Anything e2e?
most of them and specifically a storage one. You're saying we have the from above?
No, I don't see any evidence that they always end up in the default namespace (neither for TPR nor CRD). I'm not sure what that's referring to.

What I was able to reproduce is the bug described at the top of this issue: Create item in namespace, delete namespace, item remains. I observed this for TPR on and , and for CRD on .
The problem seems to be that is misspelled as in this line: If I fix that, namespace cleanup works for TPR. However, for CRD, now namespace cleanup gets stuck and controller manager logs this: Edit: To clarify, I "fixed" the check in the linked code by removing the entire highlighted block. It seems like the singular is not the only problem. It was also using the internal version of when was expected.
Regarding namespace cleanup getting stuck for CRD after the "fix", the error on the apiserver side is: According to the stack trace, the error comes from here:
yes, not only the is misspelled, but also the mismatch of the version. this is why in my original patch, I tried to ignore the version of .
I'd like to keep this open until we have a regression test since this is at least the third time we've fixed this. I'll add a test as part of making sure this fix works for CRD.


Question: expect := api.PodList{ ListMeta: api.ListMeta{ResourceVersion: "10"}, Items: []api.Pod{ <del> {ObjectMeta: api.ObjectMeta{Name: "foo", ResourceVersion: "1"}}, </del> <ins> // We expect items to be sorted by its name. </ins> {ObjectMeta: api.ObjectMeta{Name: "bar", ResourceVersion: "2"}}, {ObjectMeta: api.ObjectMeta{Name: "baz", ResourceVersion: "3"}}, <ins> {ObjectMeta: api.ObjectMeta{Name: "foo", ResourceVersion: "1"}}, </ins> }, }

kubectl returns resources in a random order, which makes it hard to look for things when users have a lot resources. Returning resources in deterministic order should be a helpful and trivial change (hopefully). We have an issue about pagination , but that's a larger and separate issue.
We should be cause to not do this on the * namespace - I think thats a special case. No disagreement for other list calls
What is the default etcd response order? Last created or last modified? Sent from my iPhone
I think the order is by key if sort param is given. If not given, the order is by last created (maybe, but we shouldn't trust it). It seems we are not trying to get sorted result from etcd.
Today, I think it's random order in the child key space in etcd (because the node structure uses a map), so you get golang map iteration order without sort. I don't know if sort on server is any cheaper than sort on client today, so we may want to do this in rest / registry rather than by passing etcd.
Apart from performance, sorting in clients means we can also take control of how responses are sorted, so for * namespace, we can sort by namespace, then by resource ID. , Clayton Coleman wrote: Master of Electrical and Computer Engineering Carnegie Mellon University
We can always hint sort to the registry storage layer in the future, so sorting on the client now won't kill us.
Just took a closer look at our rest/registry code to implement the sort (in client), but implementation-wise, it seems much easier to pass sort to etcd (one line change). Also, namespace is a prefix on resource, so sort across all namespace with etcd key will essentially sort resources by namespace then by Name. If we are to implement sort on client side, we can either: sort package: have each list object, ie.e PodList, ControllerList, implement sort interface. (We won't do this). have ListMeta implement sort interface, if there is a way to find enclosing structs (so we can use items field) have api.Object implement sort interface, verify object is list before sorting. sort algorithm ourself, and do it in rest. Is there other golang or kubernetes idom that I haven't been aware of that can be applied to this use case?

----- Original Message ----- I think sorting it on etcd is fine for now - that's the simplest possible thing that works.
Is there any material perf impact in etcd to enable it? Sent from my iPhone
Probably not material. Can always switch away if necessary
Famous last words :-) Sent from my iPhone
Got my hat ready


Question: searches = []string{} for _, s := range fields[1:] { if s != "." { <del> s = strings.TrimSuffix(s, ".") </del> <ins> searches = append(searches, strings.TrimSuffix(s, ".")) </ins> } <del> searches = append(searches, s) </del> } } if fields[0] == "options" {

This one involves systemd, DNS, musl, and Kubelet v1.25 Starting in v1.25.0, pods running on Kubelets on certain nodes with a FQDN hostname (more on that later) will not be able to resolve DNS queries if they're using musl (e.g. alpine:3.16.2, busybox, etc). Our story starts with systemd. Back in 2020, systemd a "search ." in stub resolvers if it detected the hostname to be a FQDN (to solve a separate problem). Check for a static hostname with . In , a change was made to Kubelet to propagate that "search ." into pods (to solve a separate DNS problem). That change landed in and appears in . Unfortunatey, musl based DNS resolvers struggle with this "search ." EDIT: details in musl-based DNS resolvers (depite their other quirks) should continue to function To see if you have a suitable repro node, check your (likely a symlink to . Next, start a container (any CNI) that makes some use of musl resolvers. Confirm that starting in Kubelet v1.25.0, the container now includes the "search ." if it was present on the host. Try using musl-based DNS resolution. Note that uses its own DNS resolver, its not a reproducer. Note also that pod-to-pod traffic and outbound masquerade are fine too. This is not a masquerade problem. Edit the in the container to remove the "." and your musl problems instantly disappear. Add the dot and they're back. I've been spelunking for a while. I can confirm this is not related to a specific CNI - flannel, Calico, and Cilium all reproduce the issue. Not networking related OpenShift is not affected. In 2021, RedHat folks noticed DNS "breaks" and reverted the systemd change on their OS cc <details</details<detailsThis isn't cloud provider specific. However, the most likely cases where folks use FQDN's are on-prem imo. So you're likely to see reports originating from there.

</details<details</details<detailsThis was found while preparing Typhoon's v1.25.0 release of Kubernetes v1.25.0 </details<detailscontainerd 1.6.6 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Could either revert (which propagates "search ." into containers) or find a different solution to the empty "search " problem it was trying to solve (does OpenShift even need that fix if they already ? cc A workaournd to problem is to configure hosts to strip "search ." from ala
has a great packet capture and this has been reported to musl
/sig network /cc
impressive investigation in the linked issue , that seems to conclude that is a muslc bug, since it happens outside of Kubernetes too. Just reverting the PR in Kubernetes doesn't looks right, it fixed a known bug ... and this is really a muslc bug .... maybe always removing the single dot from ? I think that for Kubernetes we should understand first what does and means and its implications, maybe some developer can help? you know someone?
Its discussed by the musl folks what means. It does not sound trivial to implement on the musl side, its a new case they did not anticipate supporting. Unfortunately this is subtle, and folks using alpine or related ecosystem things will get bitten and eventually end up here. I agree its a musl issue, but Kubelet now propagating triggers this I think its worth noting that did not set out to propagate , it was merely trying to prevent an empty . There may be better ways to achive that.
IIUC means "try the input name without any suffix". Resolvers will do this ANYWAY, but it's implementation-defined whether they do it first or last or in parallel, so allowing lets node owners be explicit (though, curiously, it seems at least glibc may try the same name AGAIN at the end of the search list).

This is pretty clearly a bug in MUSL, but I wonder if, instead of , we should have treated as rather than or the errant ? It nullifies the control that provides, but it wouldn't tickle musl like this.
/cc I foresee a new wave of bug reports caused because of this muslc bug , better to keep CoreDNS friends in the loop
The patch for this got merged for the upcoming 1.25.1 and 1.26.


Question: embed = [":go_default_library"], deps = [ "//staging/src/k8s.io/api/core/v1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/api/meta:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library",

What happened: When sending a request to the server side apply endpoint (), if the apiVersion provided in the object is different from the apiVersion in the url, it returns a 500 error, like: What you expected to happen: The request should fail if the apiVersion provided in the object is different from the apiVersion in the url, but it should fail before trying to merge, and be a 4xx error. Probably a bad request error. Anything else we need to know?: The check should probably be done somewhere near this . The issue, though, is that the patch is given as a byte array, and needs to be deserialized into an object to access the apiVersion field. How to reproduce it (as minimally and precisely as possible): With a kube-apiserver built from head, running with :
/kind bug /sig api-machinery /wg apply /priority important-longterm /assign


Question: } func TestUnregister(t *testing.T) { <ins> m1 := NewMockNoopPluginFactory() m2 := NewMockNoopPluginFactory() </ins> tests := []struct { name             string registry         Registry

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Below are the failures while running unit test with golang in master branch. Reference : $ go version go version devel go1.17- Thu Apr 22 01:40:02 2021 +0000 linux/ppc64le Note: Occurs in x86 architecture as well. The tests should Pass as with stable go git clone cd kubernetes make test KUBERACE=-race WHAT=./pkg/scheduler/framework/runtime/ Master branch of kubernetes repo. OS: Linux Observed failure on ppc64le and x86 architectures Below is the attached log:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I request help on getting this triaged. Is there a procedure to follow to get this accepted as a bug?
The Go compiler developer has identified the bugs in the test that is causing it to fail. They are listed in the last comment in the Go issue before closing it.
So the correct way to compare function would be something like would you test it in your env?
The issue still exists. Can you please reopen?
I am not sure if you can access this link for our CI job. Can you please check? https://prow.ppc64le- The above tests have been failing every day in our jobs!
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
downloads go from url="${GOLANGVERSION}.linux-" downloads go from "curl -X GET http://build-?project=build_commit -o " I suspect the postsubmit-master-golang-kubernetes-unit-test-ppc64le failed for wrong golang version.

failed for other problems. The TestMerge is not failed. For ppc64le failures, there are several issues in golang: (fixed yesterday)
Yes the go versions for both the jobs are different and it is intentional: : It is meant to run Kubernetes unit tests with the stable version of golang : It is meant to run Kubernetes unit tests with golang built from the master branch of As I mentioned in the description of this issue, these unit test cases are failing only on golang that is built from master branch and passing with stable versions of golang. And as Lynn mentioned in , the golang team has concluded that these failures are due to the bugs in the test case. And the golang issues that you mentioned do not seem to be related to the failures mentioned in this issue.
With , is false Per , do you mean that we should change to ?
No. In general comparing is the right thing to do. But if the Value is actually a , which is not comparable, comparing will panic.
/sig scheduling
Note from golang team about this bug is here:


Question: sleep 5 <del> POD_LIST_1=$($CLOUDCFG -json list pods | jq ".items[].id") </del> <ins> POD_LIST_1=$($CLOUDCFG '-template={{range.Items}}{{.ID}} {{end}}' list pods) </ins> echo "Pods running: ${POD_LIST_1}" $CLOUDCFG stop redisSlaveController

We have an e2e test that runs the guestbook example. We should also have one that runs through the other example.


Question: }) } } <ins> func TestJobBackoffOnRestartPolicyNever(t *testing.T) { jobConditionFailed := batch.JobFailed testCases := map[string]struct { // job setup parallelism  int32 completions  int32 backoffLimit int32 // pod setup activePodsPhase v1.PodPhase activePods      int32 failedPods      int32 // expectations isExpectingAnError      bool jobKeyForget            bool expectedActive          int32 expectedSucceeded       int32 expectedFailed          int32 expectedCondition       *batch.JobConditionType expectedConditionReason string }{ "not enough failures with backoffLimit 0 - single pod": { 1, 1, 0, v1.PodRunning, 1, 0, false, true, 1, 0, 0, nil, "", }, "not enough failures with backoffLimit 1 - single pod": { 1, 1, 1, "", 0, 1, true, false, 1, 0, 1, nil, "", }, "too many failures with backoffLimit 1 - single pod": { 1, 1, 1, "", 0, 2, false, true, 0, 0, 2, &jobConditionFailed, "BackoffLimitExceeded", }, "not enough failures with backoffLimit 6 - multiple pods": { 2, 2, 6, v1.PodRunning, 1, 6, true, false, 2, 0, 6, nil, "", }, "too many failures with backoffLimit 6 - multiple pods": { 2, 2, 6, "", 0, 7, false, true, 0, 0, 7, &jobConditionFailed, "BackoffLimitExceeded", }, } for name, tc := range testCases { t.Run(name, func(t *testing.T) { // job manager setup clientset := clientset.NewForConfigOrDie(&restclient.Config{Host: "", ContentConfig: restclient.ContentConfig{GroupVersion: &schema.GroupVersion{Group: "", Version: "v1"}}}) manager, sharedInformerFactory := newControllerFromClient(clientset, controller.NoResyncPeriodFunc) fakePodControl := controller.FakePodControl{} manager.podControl = &fakePodControl manager.podStoreSynced = alwaysReady manager.jobStoreSynced = alwaysReady var actual *batch.Job manager.updateHandler = func(job *batch.Job) error { actual = job return nil } // job & pods setup job := newJob(tc.parallelism, tc.completions, tc.backoffLimit) job.Spec.Template.Spec.RestartPolicy = v1.RestartPolicyNever sharedInformerFactory.Batch().V1().Jobs().Informer().GetIndexer().Add(job) podIndexer := sharedInformerFactory.Core().V1().Pods().Informer().GetIndexer() for _, pod := range newPodList(tc.failedPods, v1.PodFailed, job) { podIndexer.Add(&pod) } for _, pod := range newPodList(tc.activePods, tc.activePodsPhase, job) { podIndexer.Add(&pod) } // run forget, err := manager.syncJob(testutil.GetKey(job, t)) if (err != nil) != tc.isExpectingAnError { t.Errorf("unexpected error syncing job. Got %#v, isExpectingAnError: %vn", err, tc.isExpectingAnError) } if forget != tc.jobKeyForget { t.Errorf("unexpected forget value. Expected %v, saw %vn", tc.jobKeyForget, forget) } // validate status if actual.Status.Active != tc.expectedActive { t.Errorf("unexpected number of active pods. Expected %d, saw %dn", tc.expectedActive, actual.Status.Active) } if actual.Status.Succeeded != tc.expectedSucceeded { t.Errorf("unexpected number of succeeded pods. Expected %d, saw %dn", tc.expectedSucceeded, actual.Status.Succeeded) } if actual.Status.Failed != tc.expectedFailed { t.Errorf("unexpected number of failed pods. Expected %d, saw %dn", tc.expectedFailed, actual.Status.Failed) } // validate conditions if tc.expectedCondition != nil && !getCondition(actual, *tc.expectedCondition, tc.expectedConditionReason) { t.Errorf("expected completion condition. Got %#v", actual.Status.Conditions) } }) } } </ins>

What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): $oc create job -f $ oc get job jobname -ojson | jq .status { "active": 1, "failed": 1, "startTime": "2020-07-30T06:45:37Z" } { "active": 1, "failed": 2, "startTime": "2020-07-30T06:45:37Z" } { "active": 1, "failed": 3, "startTime": "2020-07-30T06:45:37Z" } { "active": 1, "failed": 4, "startTime": "2020-07-30T06:45:37Z" } { "conditions": [ { "lastProbeTime": "2020-07-30T06:53:10Z", "lastTransitionTime": "2020-07-30T06:53:10Z", "message": "Job has reached the specified backoff limit", "reason": "BackoffLimitExceeded", "status": "True", "type": "Failed" } ], "failed": 5, "startTime": "2020-07-30T06:45:37Z" } $ oc get job jobname -ojson | jq .spec.backoffLimit 6 Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: 4.5.3 Server Version: 4.4.13 Kubernetes Version: v1.17.1+ Cloud provider or hardware configuration: OS (e.g: ): NAME="Red Hat Enterprise Linux CoreOS" VERSIONID="4.4" OPENSHIFTVERSION="4.4" RHELVERSION="8.2" ID="rhcos" IDLIKE="rhel fedora" ANSICOLOR="0;31" CPENAME="cpe:/o:redhat:enterpriselinux:8::coreos" HOMEURL="" BUGREPORTURL="" REDHATBUGZILLAPRODUCT="OpenShift Container Platform" REDHATBUGZILLAPRODUCTVERSION="4.4" REDHATSUPPORTPRODUCT="OpenShift Container Platform" REDHATSUPPORTPRODUCTVERSION="4.4" OSTREEVERSION='44.82.-0' Kernel (e.g.

): Linux ip-10-0-192-232 4.18.0-193.13.2.el82.x8664 SMP Mon Jul 13 23:17:28 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Others: apiVersion: batch/v1 kind: Job metadata: name: sre-build-test-fail namespace: openshift-build-test spec: backoffLimit: 6 completions: 1 parallelism: 1 template: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: matchExpressions: key: node- operator: In values: infra containers: command: /bin/bash -c | # ensure we fail if something exits non-zero set -o errexit set -o nounset set -o pipefail exit 1 imagePullPolicy: Always name: sre-build-test restartPolicy: Never terminationGracePeriodSeconds: 30 tolerations: effect: NoSchedule key: node- operator: Equal value: infra
/sig apps
not sure but that is the best I can think of at this time
i have noticed this problem also, and sometimes, it seems to work normally.
so I found out that in the test is so this can cause the issue.. it has been there from the beginning of the codes inception.. WDYT?
/assign
Duplicate of . /close
Closing this issue. <detailsIn response to : Duplicate of . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: stop: true, }, { <ins> code: http.StatusOK, err:  fmt.Errorf("some error"), stop: false, }, { </ins> code: 399, stop: true, },

What happened: In a StatefulSet where each Pod has its own persistent volume provisioned by , it occasionally takes up to 30 min to detach and re-attach those disks to their node during a rollout. The following Pod events repeat: A subset of the controller-manager's logs (full logs on demand): State of the disk few minutes before the attach operation succeeds, returned by the Azure API: In this particular VMSS, each instance has at most 2 disks attached. In busy clusters, a node can have up to 5 disks attached. The operation eventually succeeds, ~20m later in this case: What you expected to happen: Disks take no longer than 3 min to re-attach (the typical time it takes in our customers' environments). How to reproduce it (as minimally and precisely as possible): Here is a copy of the manifest of the affected StatefulSet: <details<summarysolr-</summary</details</brStorage class <details<summarydefault-</summary</details</brAnything else we need to know?: Environment: Kubernetes version: v1.12.7 Cloud provider: Azure (via AKS-Engine) OS (e.g: ): Ubuntu 16.04.6 LTS Kernel: 4.15.0-1042-azure Install tools: Network plugin and version: Others: observed in westeurope and eastus regions paging /sig azure
can you also provide following info on master node: How many disks are on one VM? We are experiencing slow disk attach/detach issue when disk num is large:
I just the info about disks to the original message: between 2 and 5 per VMSS instance. The command you posted returns nothing, but if I remove "vm" from the last grep I get: Feel free to catch up on Slack if you want to have a real-time chat.
/assign paste some info here: the 2 disk attach/detach operations costs s totally, that's reason why it's stuck in so long time. BTW, there is metrics calculation issue here: , but anyway the two operations cost too long time in vmss API call level.
this seems to be an extreme case. In other environments where the issue occurs, API calls cost between 15 and 60 sec on average.

API calls cost between 15 and 60 sec on average and still takes up 30min to re-attach? could it repro easily?
Only in selected clusters unfortunately (about 5%), but when it happens it happens quite consistently. I haven't been able to reproduce outside of clusters belonging to live customers, which makes the issue a bit tricky to troubleshoot at the moment. I'm sitting in front of a cluster where it's currently happening. A Pod has been reporting for 53 min. Yesterday I observed it in the same cluster for 2 hours. We eventually deleted the Pod to unblock the situation (which doesn't always work btw). The following Pod events repeat over and over again: The disk has and just like in my first message. The controller-manager metrics also don't tell us much (14h uptime):
could you provide detailed steps about how to repro this issue? I tried to drain statefulset pods from one node to another on k8s v1.12.7, cannot repro.
The correct way to get disk attach/detach metrics on k8s v1.12.7 is like this The rolling update would cause k8s to detach disk first and then attach disk after, so I will still get following error in my testing env, but finally it could recover:
This one is a patched 1.10, with your retry fix. I have some metrics from an affected 1.12 cluster, but like I said, nothing very useful: We only have one customer on 1.12 so I'm waiting for the next incident to get back to you on a proper 1.12 setup. My understanding of what happens in most (if not all) cases is: Imagine this simplified setup 2 nodes 3 Pods belonging to a Deployment "commerce" 3 Pods belonging to a Deployment "api" 3 Pods with Azure Disks belonging to a StatefulSet "solr" We update both Deployments and the StatefulSet simultaneously (we have an operator that actually does exactly this). Both and terminate before . This time and are both scheduled on . is still Terminating. When is finally down, is already packed, so the scheduler has no other choice but to move it to . And then we are in that mode where the disk is detached, but never re-attached.

It shouldn't be any different from a drain, but at least it can maybe help you understand our typical repro scenario. We recently optimized scheduling to avoid that schenario where stateful Pods must be relocated, but other events can cause relocation like nodes failures, autoscaling, ...
here is the conclusion about what we found now: Periodically k8s cluster failed to attach a disk to a VMSS instance, that disk is actually in “unattached” state, it's caused by vmss instance in unhealthy state, although by azure cli command , the instance shows healthy. workaround: run to update the VMSS instance manually, it could recover in a minute or so We are asking VMSS team to find the root cause of this issue. In brief, if you find k8s could not attach an state disk to a VMSS node, it's probably due to the vmss instance issue, run to update the VMSS instance manually could solve this issue.


Question: return options, fmt.Errorf("service-cluster-ip-range[0] is not a valid cidr") } <del> serviceIPRange, apiServerServiceIP, err = master.DefaultServiceIPRange(*(primaryServiceClusterCIDR)) </del> <ins> serviceIPRange, apiServerServiceIP, err = master.ServiceIPRange(*(primaryServiceClusterCIDR)) </ins> if err != nil { return options, fmt.Errorf("error determining service IP ranges for primary service cidr: %v", err) }

API server defaults the service CIDR. We should instead just fail if it is not specified. Defaulting is a wild guess and very likely to be wrong. /sig network /kind bug
For existing clusters that don't currently set it, is setting it to anything other than the current default a breaking change?
Yes. You can not really change it (well, you can expand it, but not totally move it), unless you delete ALL services, including kubernetes.default. I filed this aspirationally. It's not on fire, it's just a dumb thing to have provided a default for.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kubernetes_test import ( "context" "net/http" "net/http/httptest" "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes" "k8s.io/client-go/kubernetes/scheme" "k8s.io/client-go/rest" ) func TestClientUserAgent(t *testing.T) { tests := []struct { name      string userAgent string expect    string }{ { name:   "empty", expect: rest.DefaultKubernetesUserAgent(), }, { name:      "custom", userAgent: "test-agent", expect:    "test-agent", }, } for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { userAgent := r.Header.Get("User-Agent") if userAgent != tc.expect { t.Errorf("User Agent expected: %s got: %s", tc.expect, userAgent) http.Error(w, "Unexpected user agent", http.StatusBadRequest) return } w.Header().Set("Content-Type", "application/json") w.Write([]byte("{}")) })) ts.Start() defer ts.Close() gv := v1.SchemeGroupVersion config := &rest.Config{ Host: ts.URL, } config.GroupVersion = &gv config.NegotiatedSerializer = scheme.Codecs.WithoutConversion() config.UserAgent = tc.userAgent config.ContentType = "application/json" client, err := kubernetes.NewForConfig(config) if err != nil { t.Fatalf("failed to create REST client: %v", err) } _, err = client.CoreV1().Pods("").List(context.TODO(), metav1.ListOptions{}) if err != nil { t.Error(err) } _, err = client.CoreV1().Secrets("").List(context.TODO(), metav1.ListOptions{}) if err != nil { t.Error(err) } }) } } </ins>

It looks like kubelet started using generic user agent like "Go-http-client/2.0" and makes it harder to attribute the request to kubelet. In the past we have used "kubelet" user-agent which was providing some value. Sample logs: Kubelet's requests are using customised user-agent Run kubelet Check kube-apiserver's logs: No response master tested on gce <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The default user agent name is generated like below. I am using v1.23.4 and the user agent can show kubelet as the user agent.(not on gce)
/cc We should identify those and set the user-agent correctly , is right, it is very useful field ... but I see ~kubectl~ kubelet user-agents in the log linked
Interesting... in I see both kubelet user agent and "Go-http-client/2.0": Moreover, it looks like "kubelet" user agent is used only for events-related calls and other (like leases) are using "Go-http-client/2.0".
interesting , the user agent is to the request by wrapping the roundtripper, that is when the client is generated the clientset creation is here Speculating, I'm wondering if the updateTransport is mutating the transport and not adding the corresponding wrapper in ?
/assign I think that this was my fault, let me check it in more detail
I think that this was the problem


Question: <del> DOCKER_OPTS="--bridge cbr0 --iptables=false -r=false" </del> <ins> DOCKER_OPTS="--bridge cbr0 --iptables=false --ip-masq=false -r=false" </ins>

We need to version-lock to docker 1.2 until they fix this flag, or else we need to drop 1.2 support, or else we need to detect the docker version before writing the flags file. - we need something here ASAP.
After a little bit thinking, I think we shouldn't have the problem. kube-push only updates our own binaries, not docker's. kube-up builds a new cluster which will pull the latest docker, but we can change docker's defaults, so that at the minion's provision stage, we disable both --iptables and --ip-masq.
Something like for gce.


Question: # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> liggitt </del> pwittrock seans3 soltysh

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the sample-cli-plugin repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig cli

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the cli-runtime repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig cli


Question: // or can be started, and updates the cached pod state so that downstream components can observe what the // pod worker goroutine is currently attempting to do. If ok is false, there is no available event. If any // of the boolean values is false, ensure the appropriate cleanup happens before returning. <ins> // // This method should ensure that either status.pendingUpdate is cleared and merged into status.activeUpdate, // or when a pod cannot be started status.pendingUpdate remains the same. Pods that have not been started // should never have an activeUpdate because that is exposed to downstream components on started pods. </ins> func (p *podWorkers) startPodSync(podUID types.UID) (ctx context.Context, update podWork, canStart, canEverStart, ok bool) { p.podLock.Lock() defer p.podLock.Unlock()

The following log line is often (~30times) logged during our tests in : . This suggests a possible bug as it was assumed to do not happen. No log line saying . Look at artifacts of all (or almost all) builds of: No response <details</details<detailsReproducible in tests </details<details</details<details</details<details</details<details</details
/sig node FYI
/triage accepted since it's a new error, it is a regression. So adding a corresponding priority /priority important-soon
are you working on this? Should I assign bug to you?
Yes, I'm taking a look at this. /assign
/cc
This is a metrics only error (we don’t report a fraction of metrics) and has no functional impact.
That makes sense, but is it expected that this is a valid state in first place? . My assumption was that if static is false, (i.e. API pod) then we should have a config?
I took a closer look at the logs, it seems that this is happening when we enter while a static pod is being synced which has UID reuse and another pod which has the same full name did not terminate yet. In this case, we block the static pod from starting i.e. () returns false: Because the pod can't start, we don't set : which means that when we log metrics there is no set and thus or are both defaulted to false:
Based on the above, it looks like this case is expected. As a result, maybe the simplest option is to simply drop the log statement since by design we only want to into account a subset of the states for the metrics?


Question: err = framework.WaitForClusterSize(c, int(replicas-1), 10*time.Minute) Expect(err).NotTo(HaveOccurred()) <del> By("waiting for podGC to remove/recreate any pods scheduled on the now non-existent node") framework.WaitForPodAddition(c, ns, 2*time.Minute) </del> <ins> By("waiting 1 minute for the watch in the podGC to catch up, remove any pods scheduled on " + "the now non-existent node and the RC to recreate it") time.Sleep(time.Minute) </ins> By("verifying whether the pods from the removed node are recreated") err = framework.VerifyPods(c, ns, name, true, replicas)

https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
https://k8s- Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite}
can you please take a look at this test or reassign it to someone. It is becoming much more frequent and is failing the serial test suite preventing me from finding a green build to branch for 1.5.
Looks like you've touched this test in the past. Could you take a look at it why it is flaking so frequently now?
This is likely due to the NC changes introduced in I remember having to touch that test. I'll take a look at this today.
Thank you!

I'm trying to cut the 1.5 branch and I can't find a green build so the sooner this can get fixed the better!


Question: rc=0 # List of dependencies we need to avoid dragging back into kubernetes/kubernetes <del> forbidden_repos=( "k8s.io/klog"  # we have switched to klog v2, so avoid klog v1 ) for forbidden_repo in "${forbidden_repos[@]}"; do deps_on_forbidden=$(go mod graph | grep " ${forbidden_repo}@" || echo "") if [ -n "${deps_on_forbidden}" ]; then kube::log::error "The following have transitive dependencies on ${forbidden_repo}, which is not allowed:" echo "${deps_on_forbidden}" echo "" rc=1 fi done </del> <ins> # Check if unwanted dependencies are removed go run k8s.io/kubernetes/cmd/dependencyverifier "${KUBE_ROOT}/hack/unwanted-dependencies.json" </ins> outdated=$(go list -m -json all | jq -r " select(.Replace.Version != null) |

Add a file to track dependencies we want to remove or avoid picking back up. Examples: (as opposed to ) {analysis,spec,strfmt,validate} () and () () Add a presubmit that uses to do the following: if any items in the "unwanted but still present" list are no longer present and prompts to move them to the "absent and not allowed" list if any items in the "absent and not allowed" list are present and fails. Dependency updates often reintroduce problematic dependencies that were previously dropped. /area code-organization /cc
/sig architecture /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig architecture /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I may help with it. /assign I checked the code, there is already a logic to check forbidden dependencies. can draw a go mod graph
yeah, that was sort of ad-hoc, and doesn't do the ratcheting bit we want (record a dependency we have but don't want, once we no longer use it, require moving it to the forbidden list)


Question: <ins> apiVersion: test.com/v1 kind: Example metadata: name: test spec: test: test </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Changing the scope field of a CRD ist not possible since it's immutable. But even after deleting and re-creating the CRD with changed scope it doesn't let me create resources of the CRD's kind. Re-creating CRD with changed scope should just work. Here is a full example that in the end leads to the error when trying to create a resource after re-creating the CRD. Kubernetes version (use ): v1.21.4 Cloud provider or hardware configuration: vSphere 7.0 OS (e.g: ): Centos 7 Kernel (e.g. ): 3.10.0-1160.42.2.el7.x86_64
/sig api-machinery
/assign /triage accepted
This appears to be working as intended. The example custom resource you use: It Cluster scoped because it has no namespace. But once the CRD is namespace scoped, you should not be able to create cluster scoped custom resources of that type. If instead you add a namespace to Example, it should be created, e.g.:
if the above seems to have answered your question - please feel free to close the issue :) Thank you!
kubectl will automatically use the namespace so this is definitely not the issue here. I can reproduce this issue with my above script in a fresh (v1.22.2) cluster. You can add the namespace in the last manifest, it makes no difference. From my understanding this really seems to be a bug.
Yes, that's true. Good catch. Okay, we need to root cause this.
I tried using curl to create the custom resource directly against the API and got the same result: So the problem seems to be in the API server (not in kubectl).
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: err := ds.checkpointManager.GetCheckpoint(id, checkpoint) if err != nil { glog.Errorf("Failed to retrieve checkpoint for sandbox %q: %v", id, err) <ins> if err == errors.ErrCorruptCheckpoint { err = ds.checkpointManager.RemoveCheckpoint(id) if err != nil { glog.Errorf("Failed to delete corrupt checkpoint for sandbox %q: %v", id, err) } } </ins> continue } result = append(result, checkpointToRuntimeAPISandbox(id, checkpoint))

/kind bug are failing. seems like the likely cause. cc /assign
ping
seems one test still failing.


Question: FilenameParam(flags.Filenames...). Flatten(). Do() <ins> checkErr(r.Err()) </ins> count := 0 err = r.Visit(func(info *resource.Info) error {

I've reported this before.
From a Makefile I get exit status 255.
Just to be clear (sorry for the basic question), are you saying that when the service file is malformed the crash is unhelpful for debugging? Or that creating any/all services using is currently broken?
The YAML file is not malformed. I think there was something perhaps not quite right about firewalls/rules etc. and this caused to crash unhelpfully.