Question: "k8s.io/cli-runtime/pkg/genericiooptions" "k8s.io/cli-runtime/pkg/printers" "k8s.io/cli-runtime/pkg/resource" <ins> "k8s.io/klog/v2" </ins> cmdutil "k8s.io/kubectl/pkg/cmd/util" "k8s.io/kubectl/pkg/drain" "k8s.io/kubectl/pkg/scheme"

kubedrain has "onPodDeletedOrEvicted" callback that can be used in custom controllers. It would be very helpful if we add more callbacks that can give more granular control over monitoring, like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed". It would help monitor the draining of nodes on a more granular level. Sometimes while draining, if the pod is stuck during the eviction or deletion of node, there is not much information that custom controllers can get from kubedrain, with the above-mentioned callbacks custom controllers will know which pod is evicting or if it failed during the eviction process.
/sig node
What is the custom controllers in your scenario? Can you elaborate on what you want to achieve with more details. Is this is kubectl improvement, the best sig to involve is sig cli. /sig cli
/remove-sig node please re-apply the sig node if this will end up being a request to sig node
In our case we have in CAPI (). Sometimes while draining, if the pod or volume is stuck during the eviction, there is not much information that we get from kubedrain.Helper. It would be much easier for the controller to report these kinds of issues with more granular callbacks like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed" on kubedrain.Helper.
/assign Assigning this to myself, I have discussed this issue in sig-cli community meeting and they have given green light for this improvement.
/triage accepted /priority backlog


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package v1beta2 import ( "fmt" apps "k8s.io/api/apps/v1beta2" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" ) // DeploymentListerExpansion allows custom methods to be added to // DeploymentLister. type DeploymentListerExpansion interface { GetDeploymentsForReplicaSet(rs *apps.ReplicaSet) ([]*apps.Deployment, error) } // DeploymentNamespaceListerExpansion allows custom methods to be added to // DeploymentNamespaceLister. type DeploymentNamespaceListerExpansion interface{} // GetDeploymentsForReplicaSet returns a list of Deployments that potentially // match a ReplicaSet. Only the one specified in the ReplicaSet's ControllerRef // will actually manage it. // Returns an error only if no matching Deployments are found. func (s *deploymentLister) GetDeploymentsForReplicaSet(rs *apps.ReplicaSet) ([]*apps.Deployment, error) { if len(rs.Labels) == 0 { return nil, fmt.Errorf("no deployments found for ReplicaSet %v because it has no labels", rs.Name) } // TODO: MODIFY THIS METHOD so that it checks for the podTemplateSpecHash label dList, err := s.Deployments(rs.Namespace).List(labels.Everything()) if err != nil { return nil, err } var deployments []*apps.Deployment for _, d := range dList { selector, err := metav1.LabelSelectorAsSelector(d.Spec.Selector) if err != nil { return nil, fmt.Errorf("invalid label selector: %v", err) } // If a deployment with a nil or empty selector creeps in, it should match nothing, not everything. if selector.Empty() || !selector.Matches(labels.Set(rs.Labels)) { continue } deployments = append(deployments, d) } if len(deployments) == 0 { return nil, fmt.Errorf("could not find deployments set for ReplicaSet %s in namespace %s with labels: %v", rs.Name, rs.Namespace, rs.Labels) } return deployments, nil } </ins>

As a prerequisite of moving the , , , and controllers to use the apps/v1beta2 APIs, we need to add list expansions for , , , and to .
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps


Question: <ins> apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: examples.test.com spec: group: test.com scope: Namespaced versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: test: type: string names: plural: examples singular: example kind: Example </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Changing the scope field of a CRD ist not possible since it's immutable. But even after deleting and re-creating the CRD with changed scope it doesn't let me create resources of the CRD's kind. Re-creating CRD with changed scope should just work. Here is a full example that in the end leads to the error when trying to create a resource after re-creating the CRD. Kubernetes version (use ): v1.21.4 Cloud provider or hardware configuration: vSphere 7.0 OS (e.g: ): Centos 7 Kernel (e.g. ): 3.10.0-1160.42.2.el7.x86_64
/sig api-machinery
/assign /triage accepted
This appears to be working as intended. The example custom resource you use: It Cluster scoped because it has no namespace. But once the CRD is namespace scoped, you should not be able to create cluster scoped custom resources of that type. If instead you add a namespace to Example, it should be created, e.g.:
if the above seems to have answered your question - please feel free to close the issue :) Thank you!
kubectl will automatically use the namespace so this is definitely not the issue here. I can reproduce this issue with my above script in a fresh (v1.22.2) cluster. You can add the namespace in the last manifest, it makes no difference. From my understanding this really seems to be a bug.
Yes, that's true. Good catch. Okay, we need to root cause this.
I tried using curl to create the custom resource directly against the API and got the same result: So the problem seems to be in the API server (not in kubectl).
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cache import ( "k8s.io/utils/clock" "time" ) type RetryWithDeadline interface { After(error) ShouldRetry() bool } type retryWithDeadlineImpl struct { firstErrorTime   time.Time lastErrorTime    time.Time maxRetryDuration time.Duration minResetPeriod   time.Duration isRetryable      func(error) bool clock            clock.Clock } func NewRetryWithDeadline(maxRetryDuration, minResetPeriod time.Duration, isRetryable func(error) bool, clock clock.Clock) RetryWithDeadline { return &retryWithDeadlineImpl{ firstErrorTime:   time.Time{}, lastErrorTime:    time.Time{}, maxRetryDuration: maxRetryDuration, minResetPeriod:   minResetPeriod, isRetryable:      isRetryable, clock:            clock, } } func (r *retryWithDeadlineImpl) reset() { r.firstErrorTime = time.Time{} r.lastErrorTime = time.Time{} } func (r *retryWithDeadlineImpl) After(err error) { if r.isRetryable(err) { if r.clock.Now().Sub(r.lastErrorTime) >= r.minResetPeriod { r.reset() } if r.firstErrorTime.IsZero() { r.firstErrorTime = r.clock.Now() } r.lastErrorTime = r.clock.Now() } } func (r *retryWithDeadlineImpl) ShouldRetry() bool { if r.maxRetryDuration <= time.Duration(0) { return false } if r.clock.Now().Sub(r.firstErrorTime) <= r.maxRetryDuration { return true } r.reset() return false } </ins>

When etcd is unable to elect leader for 3 election cycles, etcd returns "no leader" error through watches. This can be seen in api-server logs as . Also, when apiserver receives this error, it terminates all watchers: . In large clusters, this can destabilize master, as all these watchers need to be recreated almost at the same time, overwhelming api-server quite often. I propose adding new api-server flag that allows to specify timeout, during which we are retrying to reestablish apiserver <-etcd watches, without terminating all api-server watchers. If we are unable to reestablish watches within this specified timeout, we start terminating all watchers. Losing etcd leader for ~2-3s does not destabilize api-server. Trigger prolonged reelection of etcd leader. No response Observed in all kubernetes versions 1.19-1.24 N/A <details</details<details</details<details</details<details</details
/sig scalability /sig api-machinery /priority important-soon
/wg reliability
/assign
/triage accepted
indeed, this is a source of instability. Is this coming from ?
I believe it's coming from this change. As I mentioned, the idea to mitigate it is to add some parameter that will allow us to ignore this error coming from etcd for specific duration without terminating all watchers, for example 10s be default.
With this comment thread in mind: It would probably be cleaner to allow for simply setting the "timeout" as part of WithRequireLeader option (instead of using the constant of 3). But that would require etcd changes, including the API and we want to workaround that in much smaller timeframe. But a comment about how to make that cleaner with some additional context from here would help.


Question: } // NewProxierHealthServer returns a proxier health http server. <del> func NewProxierHealthServer(addr string, healthTimeout time.Duration, recorder record.EventRecorder, nodeRef *v1.ObjectReference) *ProxierHealthServer { </del> <ins> func NewProxierHealthServer(addr string, healthTimeout time.Duration, recorder record.EventRecorder, nodeRef *v1.ObjectReference) ProxierHealthUpdater { </ins> return newProxierHealthServer(stdNetListener{}, stdHTTPServerFactory{}, clock.RealClock{}, addr, healthTimeout, recorder, nodeRef) } <del> func newProxierHealthServer(listener listener, httpServerFactory httpServerFactory, c clock.Clock, addr string, healthTimeout time.Duration, recorder record.EventRecorder, nodeRef *v1.ObjectReference) *ProxierHealthServer { return &ProxierHealthServer{ </del> <ins> func newProxierHealthServer(listener listener, httpServerFactory httpServerFactory, c clock.Clock, addr string, healthTimeout time.Duration, recorder record.EventRecorder, nodeRef *v1.ObjectReference) *proxierHealthServer { return &proxierHealthServer{ </ins> listener:      listener, httpFactory:   httpServerFactory, clock:         c,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: , from whatever combination of inputs we stuck into it, segfaulted. The segfault, which originates from a slightly modified fork of v1.17, looks as such: Translated to master, where the problem appears to be present, the segfault originates from this call: And that is called from here: This seems odd, because The only pointer involved in the segfaulting call is itself The caller nil-checks before calling a method on . I did some digging and found , which is a fix for this problem. However, this fix appears to be undone in . There doesn't seem to be a reason for this to have been changed in the latter PR. What you expected to happen: should not segfault from any bizarre combination of user input. If the situation is invalid, it should fail gracefully. How to reproduce it (as minimally and precisely as possible): Honestly, I'm not 100% sure what combination of inputs we crammed into , but I can locate them and post them if necessary. Anything else we need to know?: I don't feel confident opening a PR to fix this myself, but in addition to recreating the fix from , a comment should be explaining why there is this weird, seemingly pointless layer of indirection when assigning to the interface, so that this doesn't get un-fixed again.
Maybe decompose the above statement into multiple lines which should give us more idea which field was nil.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
can't be nil, as it is of type , which is not an interface or pointer is an interface. However, when built from the exported constructor, , it is set to a literal , which is likewise not a pointer Therefore, the only remaining field is itself. Plus, I don't quite recall my go segfault stack trace syntax, but I'm relatively sure that indicates that the first argument to the method is , nil, and that the first argument to methods is the object being called on.

/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
When config.HealthzBindAddress is empty, we would encounter the scenario depicted in the stack trace. I have proposed a fix.
/assign
That commit, , is part of the PR that I believe introduced the problem. I strongly believe the problem to be one of a non-nil interface with a nil receiver. This is the only way that could be nil in the snippet of code from . Before PR (the changes from which are present in the version I ran), there was this odd looking set of code: The variable healthzServer, which is a pointer of type , was assigned to the variable , which is an interface of type . Then, later on, the variable was passed to , as an argument of type . This looks unexpected, and it appears that the variable is extraneous. After the changes in PR , the variable was removed, and was passed directly to as the argument for interface type . The former set of changes, with the intermediate variable, are meant to avoid the situation of a non-nil interface with a nil value. If the variable is never set in the conditional statement, then it is a nil interface. If is passed directly, then the result is the argument to is an an interface who's value is and who's concrete type is . This means that is called with as the receiver. I strongly believe this to be the root cause of the issue, but I do not know enough about kubernetes to open a PR with the correct fix, which would be to reinstate the variable (after the renaming in , this would be a variable of type ) to use as a go-between, to once again avoid this situation of a non-nil interface with a nil receiver. I believe that the removal of the variable was out of a misunderstanding of its purpose. If my assessment of the problem is incorrect, please let me know so I can understand where my logic is wrong. It is very possible that I am misunderstanding the phenomenon of a non-nil interface with a nil receiver. I would open a PR myself, but I do not know enough about Kubernetes to be confident in my ability to correctly fix and test this problem.
/remove-triage unresolved


Question: var IP string // TODO: Remove this when sandbox is available on windows <ins> // Currently windows supports both sandbox and non-sandbox cases. </ins> // This is a workaround for windows, where sandbox is not in use, and pod IP is determined through containers belonging to the Pod. <del> if IP = ds.determinePodIPBySandboxID(podSandboxID); IP == "" { </del> <ins> if IP = ds.determinePodIPBySandboxID(podSandboxID, r); IP == "" { </ins> IP = ds.getIP(podSandboxID, r) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: On Windows Server with sandbox support the CNI gets called for all the containers within the pod including the sandbox container itself. What you expected to happen: The CNI should be called only for the sandbox container since they share the same network details. How to reproduce it (as minimally and precisely as possible): kubelet Pod at the parameters that get passed when the CNI is called Anything else we need to know?: This applies to Windows Server with sandbox support. The CNI should be called only for the infra container. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig windows /sig network /kind bug /assign /cc
Reopen this because of
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
has been merged now. /close
Closing this issue. <detailsIn response to : has been merged now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
has this issue been fixed? Is the CNI getting called only for the infra container? From what I see, the CNI still gets called for every single container from within a pod:
See comments It is required for windows containers.
The way I understand it, it's that the DNS issue will be fixed in Windows RS5 and there will not be required additional calls. The workaround at the moment is to call the CNI multiple times until the DNS issue gets fixed in Windows. As far as I know, that is the only reason why the additional calls to the CNI are required.
Is this true? Do we still need to invoke cni for each container in RS5? And even with new cni dns capabilities ()?

the PR that you linked it's unrelated to this issue. The CNI should be called for the sandbox pod which is the infra container. All the other containers created within the same pod are sharing all network details, excepting DNS. This is an issue in 1709 and 1803 where not everything gets replicated properly to the other containers. As a workaround for this, you need to call AttachEndpoint from HCS on the newly created container network interface only to replicate the DNS as well. You can also see more details about this issue here:
Can you re-open the issue in the meanwhile since this issue still exists?
I still think this isn't a issue, but a requirement for windows containers. Even after RS5, we should probably support old windows versions. WDYH
Why would the spec of the CNI should be different on Windows? On Linux you know that the sandbox ID that will get passed to the CNI is always the sandbox container. On Windows the sandbox ID won't be only that, it will also be an ID of a container from within the same pod. Does this sound like normal behavior to you? For now I agree, the workaround is to call the CNI for every single container because of the Windows issue with DNS. But this doesn't mean we have to stick with this for future Windows releases which will not require calling the CNI ADD multiple times. Also, this is a hack for now () which needs to be fixed. Please re-open the issue until we get feedback from others as well, thanks.
OK. Reopened the issue. Let's wait a while for feedbacks from other folks.
Apologize for the delay. Windows doenst have the namespace support native on the platform. we this support as part of RS5 Platform release. We are working on making runtime spec changes and CRI/ContainerD changes to use this model. here is a runtime spec PR that got merged. This is orthogonal to the DNS issue. Once we are done with the CRI/ContainerD changes then if u use CRI/containerD as runtime and if your OS version is RS5 and above then you don't need this second CNI add call. But as Pengfei said, if you are on a downlevel version or using dockershim as Runtime then you will have to make the second CNI call.

So in my opinion this code will stay as workaround forever until we remove the support for downlevel OS. Hope this helps. I would recommend closing this issue.
thanks for the confirmation. In my opinion this should stay open until it gets fixed at least for future Windows releases, because the way it is at the moment, the CNI will still be called multiple times regardless of the Windows version. Why would you recommend closing this issue? Shouldn't this be fixed for futures Windows releases? The other calls will be useless when Windows gets proper namespace support. Also, I recommend watching this presentation about the CNI which had place at the KubeCon from May I don't see why we shouldn't have this open until the issue gets fixed for future releases and we stick with the official SPEC of the CNI.
and I had talked before about end of life of downlevel OS like 1803 and 1709. I think we should keep this open or open a new Issue to track updating the code so that the CNI is not called multiple times
This code would exists as far as we support 1709, 1803. The new workflow to use the windows namespace is getting implemented in containerd/cri. We can move kubernetes to use this new flow only for RS5 and above, if we plan to do it here as well. There isn't a good way to switch between 2 workflows of cni callflow wrt the Windows version hence we are not planning to change here till 1709 & 1803 exists, without adding some ugly code.
thanks for the explanation
I totally agree that this has to be fixed for future release and we are actively working on it. I was recommending to close this issue and open a feature to track this request. We should be able to raise a PR in another 2 weeks to support native network namespace support. This code will stay as long as we support down-level OS'es. I am fine either way. I am trying to close any open issues that is not a bug on Windows Networking(Platform).
/kind feature Although from a kubernetes standpoint, this isn't a new feature. It doesn't need a new KEP and won't change any API. It will just be a future Windows network code cleanup.
/kind cleanup
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
- can you clarify a few things so we know what's required to fix this? 's the minimum OS version needed? Windows Server 2019, or a future release? this be possible to fix for dockershim, or just CRI-containerD? Thanks!
What's the minimum OS version needed? Windows Server 2019, or a future release? [Dinesh]: Minimum OS version needed is 2019 Will this be possible to fix for dockershim, or just CRI-containerD? [Dinesh]: yes its possible to fix for Dockershim as well. but we are starting it with ContainerD. ContainerD is already done and available for testing.
excellent. thank you Dinesh!
ping - you have logs coming in email for this. Can you help us determine if it's needed for 1.14 vs 1.15?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Ping
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Status on this?


Question: MANIFEST_IMAGE := $(PUSH_REGISTRY)/etcd <ins> # Install binaries matching base distro permissions BIN_INSTALL := install -m 0555 # Hosts running SELinux need :z added to volume mounts SELINUX_ENABLED := $(shell cat /sys/fs/selinux/enforce 2> /dev/null || echo 0) ifeq ($(SELINUX_ENABLED),1) DOCKER_VOL_OPTS?=:z endif </ins> # This option is for running docker manifest command export DOCKER_CLI_EXPERIMENTAL := enabled # golang version should match the golang version from https://github.com/coreos/etcd/releases for the current ETCD_VERSION.

Part of What happened: Attempt to run as a non-root user Get What you expected to happen: It runs How to reproduce it (as minimally and precisely as possible): /sig release /sig cluster-lifecycle /area security And as part of kubeadm v1.16 milestone: /milestone v1.16 /priority critical-urgent
/assign randomvariable /lifecycle active
/priority important-soon


Question: func main() { rootCmd := &cobra.Command{ Use:     "app", <del> Version: "2.30", </del> <ins> Version: "2.31", </ins> } rootCmd.AddCommand(auditproxy.CmdAuditProxy)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --e2e test fails with ECDSA Service Account key e2e test to succeed with ECDSA Service Account key Use ECDSA key for the service account key. Logs from the started test container: library defaults to by default as the only supported method while should be explicitly: Relevant code: On the API server side: Kubernetes version (use ): 1.21.0-rc.0 Cloud provider or hardware configuration: OS (e.g: ): Talos Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Changing the service account key type to RSA fixes the test:
this is a problem for us, as this test is part of test suite, and it fails with 1.21-rc.0 e2e tests
SIG Auth own their test cases (it's tagged in the test name), SIG Architecture own conformance. Testing owns the frameworks, CI, tooling, etc. EDIT: just clarifying and making sure the right people see it. I raised this to the conformance work group.
/assign /priority critical-urgent /milestone v1.21 because this is in the conformance suite, it must not restrict valid+supported cluster configurations
/reopen the image needs to be promoted and bumped in the e2e test
Reopened this issue. <detailsIn response to : /reopen the image needs to be promoted and bumped in the e2e test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
promote image PR:
I believe with merged this can be closed as all the fixes were done and backported to 1.21
Thanks!


Question: import ( "fmt" "math" <ins> "reflect" </ins> "testing" "k8s.io/utils/pointer"

The new scheduler performance test was introduced back in 1.18, back then, the metric it supported are all . However, along with , some metrics are updated to be , and hence it's no longer appropriate to return the first Histogram: However, it's very common (for instance Preemption) that the metrics may hold multiple histograms: Additionally, metrics and have been removed, we should update them to : /sig scheduling
/triage accepted /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale


Question: function detect-master () { KUBE_MASTER=${MASTER_NAME} if [ -z "$KUBE_MASTER_IP" ]; then <ins> # gcutil will print the "external-ip" column header even if no instances are found </ins> KUBE_MASTER_IP=$(gcutil listinstances --format=csv --sort=external-ip  --columns=external-ip --zone ${ZONE} --filter="name eq ${MASTER_NAME}"  <del> | tail -n 1) </del> <ins> | tail --lines=+2 | tail -n 1) </ins> fi if [ -z "$KUBE_MASTER_IP" ]; then <del> echo "Could not detect Kubernetes master node.  Make sure you've launched a cluster with 'kube-up.sh'" </del> <ins> echo "Could not detect Kubernetes master node.  Make sure you've launched a cluster with 'kube-up.sh'" 1>&2 </ins> exit 1 fi echo "Using master: $KUBE_MASTER (external IP: $KUBE_MASTER_IP)"

I'm following the documentation on this doc: Everything works until this command: I get this error consistently from different VMs (a Debian and a CoreOS). Maybe I have to edit /etc/hosts to add the "external-ip" with the public IP of some server? Docs don't seem clear. Thanks, Riccardo
Its failing to look up your master correctly. You can try setting KUBERNETES_MASTER=http://<your IP hearBrendan On Sep 21, 2014 5:45 AM, "Riccardo Carlesso" wrote:


Question: if (( "${found}" == "${EXPECTED_NUM_NODES}" )) && (( "${ready}" == "${EXPECTED_NUM_NODES}")); then break <ins> elif (( "${found}" > "${EXPECTED_NUM_NODES}" )) && (( "${ready}" > "${EXPECTED_NUM_NODES}")); then echo -e "${color_red}Detected ${ready} ready nodes, found ${found} nodes out of expected ${EXPECTED_NUM_NODES}. Found more nodes than expected, your cluster may not behave correctly.${color_norm}" break </ins> else # Set the timeout to ~10minutes (40 x 15 second) to avoid timeouts for 100-node clusters. if (( attempt > 40 )); then

When pushing, I noticed this weird issue: This is because in my auto-scaling group on AWS I have turned up the desired number of nodes from the environment variable's default. The script should allow for more without question, so the check should be = not ==.


Question: v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" clientset "k8s.io/client-go/kubernetes" <del> podutil "k8s.io/kubernetes/pkg/api/v1/pod" </del> <ins> "k8s.io/kubectl/pkg/util/podutils" </ins> "k8s.io/kubernetes/pkg/controller/daemon" "k8s.io/kubernetes/test/e2e/framework" )

Several test/e2e/framework/* packages have as their only dependency on k/k. The following functions are shared that way: IsPodAvailable returns true if a pod is available; false otherwise. IsPodReady returns true if a pod is ready; false otherwise. VisitContainers invokes the visitor function with a pointer to every container spec in the given pod spec with type set in mask. If visitor returns false, visiting is short-circuited. VisitContainers returns true if visiting completes, false if visiting was short-circuited. AllFeatureEnabledContainers returns a ContainerType mask which includes all container types except for the ones guarded by feature gate. The feature gate check in AllFeatureEnabledContainers is problematic. E2E tests don't know what the feature gates in the cluster are. The rest of the functions look like something that could be moved into == Having the functions in makes it clearer that the checks only depend on aspects of the public API. Reduces dependencies of e2e/framework on k/k, in case that we want to move it into staging. Prevents accidentally using checks that only work in-tree (like feature gates).
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/assign


Question: <ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package metrics </ins>

When running , I got the following errors. Did I miss anything?
This problem will show up on non-linux platforms since the test/integration/metrics has only one file and that is marked as not runnable in non-linux platforms:


Question: "errors" "fmt" "io/ioutil" <ins> "net" </ins> "os" "path/filepath" "reflect"

What happened: The is taken from without regard for the main family of the cluster. No warning or error is given if gets the wrong family. What you expected to happen: That is taken from the "main" ip family, or at the very least the POD is not started and an error is given if it's not. How to reproduce it (as minimally and precisely as possible): Use the CNI-plugin and ipam. Use a single node cluster to avoid that assign the same address to PODs in different nodes. This may be CRI-plugin dependent, I use . The PODs are assigned dual addresses which has been supported since v1.9. Note the order; IPv6 first. In a dual-stack or a single-stack cluster with main family IPv4 will get an IPv6 address even if the main family is IPv4; Anything else we need to know?: First, note that this is not a dual-stack problem. may get the wrong family in a single-stack cluster if the PODs has dual addresses. The bug may however have been introduced with the dual-stack support, before K8s may have selected the correct family, I have not checked that. This is reported and discussed in There it is considered as a CNI-plugin problem, the CNI-plugin must present addresses in the order K8s wants them. Problem is that CNI-plugins doesn't know this. The addresses takes the path; The problem can be addressed in either of these places. The current situation is that the full responsibility lies on the , it must send the addresses in the "correct order". This is not the best place. The CNI-plugins are not a K8s-only thing and to require that a "prefered-family" or something similar must be supported by CNI-plugins to be "Kubernetes compliant" should be avoided. The CRI-plugin is a K8s-only thing and a "prefered-family" can be introduced so the CRI-plugin can sort the address array before sending it to . This is however undesirable since it adds a configuration complexity, it put a responsibility on the user or installation tool to configure the "prefered-family" to all CNI-plugins now and in the future. The best is to handle this in K8s itself where the "main" family is known. Is this really necessary?

All communication works even if is not of the main family. EndpointSlices seem to get this right regardless of order. Endpoint on the other hand seem to take the (old) podIP. So perhaps; is sufficient? If really must be , then K8s should sort the array so belongs to the main family. The problem exist on all supported K8s versions and on "master". If installed in the default way the situation is; Calico always send IPv4 first Cilum always send IPv6 first I don't think you can control the order, but I have not asked. Environment: Kubernetes versions: v1.17.12, v1.18.9, v1.19.2, v1.20.0-alpha.1, master v1.20.0-alpha.1.257+ Cloud provider or hardware configuration: None OS (e.g: ): xcluster Kernel: linux-5.8.1 Install tools: None Network plugin and version: bridge, host-local v0.8.7 Others: CRI-plugin: cri-o 1.18.3
/sig network
For the record; all communication via services works perfectly fine even if the order in is "faulty" with not belonging to the main family in k8s v1.18.8.
I think this is a duplicate of CNI plugins must report multiple IPs in the order expected by the cluster. The podIPs[0] == podIP requirement is for backwards compatibility with clients that do not read podIPs. Server-side reordering causes problems with clients that continually write/reconcile until the server list exactly matches their local list.
The problem exist in single-stack K8s clusters when PODs has dual addresses (supported since K8s v1.9, i.e way before K8s dual-stack which came in v1.16) And this issue is not a duplicate of . That issue is about a problem when the CNI-plugin uses the "correct" order. This issue is about the fact that K8s puts undocumented requirements on the CNI-plugins. There is no requirement on order in the . And since this requirement is undocumented it is not supported for instance by cilium and calico. The itself should be the requirement is for backwards compatibility with clients that do not read podIPs.

Since they doesn't read podIPs the value of should not matter to them, or? I think this should be handled in K8s, but if the responsibility is pushed downhill to the CNI-plugins then two things must be done in K8s; extra requirement on "Kubernetes compliant" CNI-plugins (outside the CNI spec) must be formally documented and communicated to the CNI-plugin providers. a "faulty" CNI-plugin accidently put's an IP-address of the "wrong" family in the first item then K8s must report an error and not start the POD. Please note again that this is not a K8s dual-stack problem. In a single-stack IPv6 K8s cluster the CNI-plugins must provide an IPv6 address first in the list.
The reason podIP must match podIPs[0] is related to updates to an existing object by an old client. See for a related problem when this was not done. An old client updated an existing object, setting only the podIP field (which is the only one it was aware of). The API server needed to detect that and populate the podIPs field correctly for new clients to read. The detection is based on the mismatch between podIP and podIPs[0], and the server responds by populating . That means that new clients (aware of both fields) must set podIP and podIPs[0] to match.
So this is weird, but... dual-stack clusters do not actually have a "main family", because decisions about IP families are made by multiple components, and kubernetes clusters do not have a single source of configuration truth that applies to all components. So it is explicitly allowed to have kube-apiserver default to IPv4-primary IPv6-secondary for services, and have your network plugin default to IPv6-primary IPv4-secondary for pod IPs, and to have some nodes have a single IPv4 address and some have a single IPv6 address and some have dual-stack addresses. It's not super clear why you'd want to do that, but we decided we had to allow it, simply because there's no easy way to prevent it. Of course in scenarios where the entire cluster is being installed/deployed by a single tool, one would expect that tool to configure all of the components in a consistent way. There is no "order K8s wants them".

Kubernetes doesn't want the addresses in any order. You want them in a particular order. Therefore you must configure the CNI plugin to return them in that order. Eg, in the example above, you should reverse the two in the config. Kubernetes's interpretation of this is that if Cilium sends the IPv6 address first, then it is telling Kubernetes that Kubernetes ought to prefer the IPv6 one. It appears that the CNI spec doesn't actually say that though... It doesn't seem to say anything about the ordering of multiple IPs... So maybe that should be clarified. If the CNI authors feel that the ordering of returned addresses is not meaningful then perhaps kubelet's behavior could be changed.
Thoughts on this?
CNI doesn't make any statement about ordering of IPs. It definitely doesn't have any notion of a "primary IP". It seems that containerd itself does have some
I mean that's literally what I just said :stuckouttongue:. The question is more if that is intentional and/or changeable. It seems strange that the decision about whether to allocate IPv4 and/or IPv6 IPs is left up to the network plugin, with no input from the runtime, but the decision about what to do with IPv4 and/or IPv6 IPs is then left up to the runtime, with no input from the network plugin.
That is very good an as it should be but... The quote is from the initial commant. Please note that this is a single-stack cluster! And that has a main family even though the PODs have dual addresses.
Right; don't comment on issues just before going to bed. Ooops. I don't think we want to introduce the notion of IP ordering / or preference in to CNI for a few reasons -- It's a bit of a breaking change, depending on how you read the spec It would be awkward to push the "preferred family" from k8s all the way down through to CNI It doesn't match how Linux works - interfaces don't have primary IPs (legacy aside). So fixing this in the apisever is probably the right place? BTW, what, exactly, defines a cluster's address family? ClusterCIDR, but that's optional (as is PodCIDR). ServiceCIDR, certainly. Is there an explicit config flag?

no flag, you configure the cluster with IPv4 or IPv6 addresses, of course if you have pod with IPv4 addresses and service with IPv6 that will not work kubelet set the status.PodIPs In a single-stack cluster you don't have a main family, you only have ONE family If we don't have PodIPs and only PodIP in a single-stack cluster this will be clearly a configuration problem in the CNI,no? I personally, think that this is a configuration problem and that we created it because we pluralised PodIP without a feature gate
Yes, and this is how it looks in an IPv4 single-stack cluster ( NOT set) if the CNI-plugin has been configured for dual addresses on PODs but returns the assigned addresses in the "wrong" order (ipv6 first);
Why is requesting an IPv6 address for pods in an IPv4 cluster not a configuration error? To me it seems a bit like running an ARM image on an AMD64 machine; Kubernetes will go through some motions but will not make it work or tell you what you did wrong.
Dual addresses in PODs has been supported from K8s v1.9, i.e. long before K8s dual-stack so that is not a configuration error. In the CNI specification there is nothing about required order for addresses or "prefered family" so it is not a configuration error for the CNI-plugin either. For K8s functions the order in does not matter. But design of upgrade requires that see Problem is that K8s (kubelet) sets "podIP = podIPs[0]" without checking the family. Here some error should be in place, like "CNI-plugin not compatible with K8s" and the POD should not be started, or better, sort the addresses or remove the requirement. This is a K8s problem. CNI-plugins should not have to care about order of reported addresses at least not without a CNI specification update.
This doesn't appear to answer my question. What is the meaning of an IPv6 pod address in an IPv4 cluster?
A cluster does not have an address family. Usually it seems like it does but that's just because the administrator has configured all of the individual components in a consistent way. But most components' configuration only affects what that component does (though in some cases that will have knock-on effects).

kube-controller-manager sets based on its (assuming is set) The network plugin (indirectly) sets (which may be affected by kcm's choice of ) based on its own configuration Either kubelet or an external cloud provider sets : based on , if passed, in the non-cloud-provider case based on cloud-provider-specific configuration (or hardcoded defaults) in the cloud case kubelet sets from kube-apiserver sets based on its kube-controller-manager creates endpoints, mostly based on the IP families chosen by kube-apiserver, except in the case of headless selectorless services, where it lets the user set endpoints of whatever family it wants, because it doesn't usually know whether the cluster is single or dual stack. kube-proxy determines whether rules are actually created for IPv4 and/or IPv6 services, based on the passed in if the dual stack feature gate is disabled, and unconditionally dual stack if the feature gate is enabled the cloud provider determines whether services get IPv4 and/or IPv6 IPs, based on cloud-provider-specific config or hardcoded defaults. (EDIT: well, and probably ) This... is kind of a mess, but there's no real way to fix it centrally given the way that kubernetes works. But we could add a config option to kubelet to unambiguously specify whether it should act like it's in a SSIPv4/SSIPv6/DSIPv4Primary/DSIPv6Primary cluster, and then make assignment make use of that.
Oof. Well. Maybe it makes sense for kubelet to stop setting PodIP, and leave that decision to the apiserver? It could make that decision based on ServiceCIDR, and order the families accordingly?
No, the dual stack KEP even explicitly calls out the fact that that it doesn't work that way: There is no single source of truth for kubernetes cluster config, and thus there is no single truth about whether the cluster is IPv4, IPv6, dual-stack-IPv4-primary, or dual-stack-IPv6-primary. If we want kubelet to impose some policy on PodIPs, then we need to tell it what policy to impose, because it has no entirely-reliable way of determining what policies any other component is imposing.
That is still much better than to require this function from the CNI-plugin which is not even K8s and follows a specification.

Please note that the networking e.g services, external access, works fine whatever order are in and whatever value has. I have not found any communication problem in K8s = v1.19 caused by this. The value of seem more "cosmetic". But it looks wrong to see ipv6 addresses of PODs in an ipv4-only cluster (). The problem seem to be an unfortunate upgrade design that imposed the constraint If that can some way be relaxed this becomes a no-problem as I see it.
/triage accepted /assign
What is the material impact of this behavior?
we can not relax because that will break existing clients. a side note. Using dualstack configured CNI on a none dual stack cluster was pre dualstack hack, if it must remain in use then CNI should only report one IP not two.
what's your reasoning to change CNI rather than kubelet?
We can't just put undocumented requirements on the CNI-plugins, they are not part of K8s, they have a formal specification and they have other users beside K8s. At least K8s must formally document the extra requirements on CNI-plugins that K8s has for the CNI-plugin to be K8s compliant.
As highlighted on the discussion there is no single truth of what clusterIP family is. Also kubelet will have to depend on which is not always used by CNIs. You can have a perfectly functional cluster that does not use PodCIDRs at all (I believe Cilium for example does not use it). Also I am not saying change CNI. A well configured CNI (single family for single stack cluster) will not have this problem.
Is the latter statement a retraction of the former? If not, how can they be reconciled?
He's saying that an administrator deploying a CNI plugin in a single-stack kubernetes cluster ought to configure that CNI plugin to return a single IP address, of the family being used by the kubernetes cluster. A CNI plugin that does not allow you configure what address families it uses may not be compatible with all kubernetes clusters. Eg, a CNI plugin that only supports IPv4 cannot be used as the network plugin in a single-stack IPv6 kubernetes cluster.
So, to sum up: Kubelet will always use the first IP returned by the CNI plugin as , regardless of any other cluster configuration.

Some CNI plugins, when configured to do dual stack, do not give you a way to configure which IP comes first, meaning kubelet may pick a that is not the IP family the admin considers the cluster's primary IP family In dual-stack clusters, this does not appear to cause any actual problems, other than that the "default" pod IPs shown in "" may not be the ones the administrator wants Nothing in the current kubelet configuration is supposed to indicate whether kubelet should prefer IPv4 or IPv6 pod IPs. It would be possible to add a new config option indicating this, or we could just declare that should also affect the sorting of pod IPs, because really why wouldn't you want that? In single-stack clusters, it has been suggested that the right answer is "don't do that then"; ie, the administrator should configure the CNI plugin to only return a single IP, and if they can't do that, then that's the CNI plugin's problem, not Kubernetes's
oh, and: if the CNI plugin returns multiple IPs which are not a dual-stack pair (eg, if it returns 2 IPv4 IPs, or 1 IPv4 IP and 2 IPv6 IPs), then kubelet will fail to update the pod at all, because it assumes it can just copy the IPs returned from CNI into , but the apiserver will only accept if it is either a single IP or a dual-stack pair.
That would be excellent! Then it would not be necessary to stop supporting dual-stack pods in a single stack K8s. And Cilium users wanting to upgrade their ipv4-only cluster to dual stack would not suddenly see ipv6 addresses on a .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Just for the record, this is an actual problem when trying to deploy rook-ceph. Rook makes strong assumption about family being the same as family and just fails to create Ceph cluster when they do not match. are configured to listen only on but the advertised addresses are the corresponding service .

This result in listening only on pod IPv6 address but being advertised using the service IPv4 address.


Question: otherPods := make([]*v1.Pod, 0, len(pods)) for _, p := range pods { op := p.DeepCopy() <del> for _, c := range op.Spec.Containers { allocatedResources, found := kl.statusManager.GetContainerResourceAllocation(string(p.UID), c.Name) if c.Resources.Requests != nil && found { c.Resources.Requests[v1.ResourceCPU] = allocatedResources[v1.ResourceCPU] c.Resources.Requests[v1.ResourceMemory] = allocatedResources[v1.ResourceMemory] } } </del> <ins> kl.updateContainerResourceAllocation(op) </ins> otherPods = append(otherPods, op) } attrs.OtherPods = otherPods

After enabling In-Place Pod Vertical Scaling, if deploy a pod without setting memory (or cpu) request, kubelet will be failed at its second restart kubelet restart successfully kubelet with In-Place Pod Vertical Scaling enabled a pod without setting memory request, e.g: kubelet for the first time kubelet for the second time will fail with error: After deploying the pod, the following information is saved in the file : Then restart kubelet for the first time, the relevant content in the file becomes: When deploying the pod, as there are no records in the checkpoint (), kubelet will get resource config from directly and save it to the checkpoint (without memory request) When restarting kubelet for the first time, it will restore the previously saved data from the checkpoint. Since the memory request was not previously set, it will be set to an empty value and resaved to the checkpoint When restarting kubelet for the second time, it will process as a value of 0 (not empty), which resulting in the inconsistency between the calculated checksum during data recovery and the previous one <details</details<details</details<details</details<details</details<details</details<details</details
/sig node
cc
/triage accepted
/assign
xlink: /cc
thanks for finding this issue and the fix. It looks good with suggestion. For beta, I wonder if we can do away with local checkpoint and rely on PodStatus as persistent store and source of truth for allocated resources. It will simplify the implementation.
Using resource.Quantity in checkoutPoint is a bad usage. I found except the case in this issue, whenever adding a static pod to the kubelet, then restart the kubelet, restore the resize status checkpoint will fail, casing kubelet panic. The reason is , before restart kubelet, the checkpoint struct in kubelet will looks like: and the the podstatusmanager_state file is looks like: and after restart kubelet and recover the checkoutpoint state, the checkpoint struct in kubelet will looks like: the difference is between "kube-scheduler:map[cpu:{i:{value:6 scale:-1} d:{Dec:<nil} s: Format:DecimalSI}" and "kube-scheduler:map[cpu:{i:{value:600 scale:-3} d:{Dec:<nil} s:600m Format:DecimalSI} ".

It's caused by json Marshal and Unmarshal of resource.Quantity. So, I think Using resource.Quantity in checkoutPoint is a bad case. We should consider using PodStatus in apiserver instead of checkoutPoint.
Is this problem solved? Now I can still reproduce this problem in the official version of 1.28. Thanks~
Could anyone please let me know what is the state of the art to solve this issue?
we have seen the similar problem after enabling InPlaceUpdate pod resources and could we reopen this ?
can we just do marshal&unmarshal before storing state checkpoint to avoid the issue of resource.Quantity?
I am +1 on doing away with the checkpointing code altogether and leveraging PodStatus. We discussed this in the Kubernetes Contributor Summit in Chicago. I just have not been able to find the cycles to work on it :(


Question: if [ "${FAIL_ON_CHANGES}" = true ]; then echo "running FAIL_ON_CHANGES" <del> ret=0 if diff --ignore-matching-lines='^s*"Comment"' -NauprB -I "GoVersion.*|GodepVersion.*" "${CLIENT_REPO}" "${CLIENT_REPO_TEMP}"; then </del> <ins> # ignore base.go in diff cp "${CLIENT_REPO}/pkg/version/base.go" "${CLIENT_REPO_TEMP}/pkg/version/" if diff -NauprB  -I '^s*"Comment"' -I "GoVersion.*|GodepVersion.*" "${CLIENT_REPO}" "${CLIENT_REPO_TEMP}"; then </ins> echo "${CLIENT_REPO} up to date." cleanup exit 0

The verification tests have been since we enabled them on the release-1.6 branch. It looks like is the failing check. From : Does this check need to be fixed? Should we disable this check on release branches? Or is something actually wrong with the release branch? cc
I don't actually know what this test does. Does this mean is going to fail, or is it just a CI issue? If so that blocks today's beta cut.
cc
Sent
somehow the most recent commit () fixed this. I'm not sure what it did differently vs. the previous times it ran.
I cherrypicked the proposed fix from . It didn't work without that.
I am still getting CI errors during steps for . when I build locally the scripts copies . But when I push the PR, the verification always fails because head does not have that vendored package: and I was hoping would fix, but it is still happening. Any idea?
I don't think client-go depends on , so the verification script is correct. How did you update staging/client-go? Did you do it by running hack/update-staging- Did you rebase your PR to include latest changes on master before you ran the update script?
I was using and that introduced/copied the package. When the update-all- was introduced in HEAD, I got the same result. If I remove the package, my local build fails.
Yes, I have been rebasing consistently trying to get rid of error, but nothing helps so far. If I remove the dependency, my build fail locally and still doesn't pass the CI check.
Just rebased, removed the package, and ran the update script, this is the result:
Does client-go actually depend on ?
please try to find out which of the client-go packages depend on pborman/uuid. You can use something like this:
at some point when the staging build was enabled over the weekend a local build pulled a version of that had a dependency on package. I finally figured out to delete that client-go and got a clean build without that package dependency. Thanks.
where was that faulty client-go? In the GOPATH? Can we do anything to avoid those problems in the future?
it was in the same dir the kubernetes project. Not sure when it got pulled down.

I really never used the staging script until earlier this week. So I was not too familiar with its behavior. Once I started to research here in this thread and went back and read the README for staging, I suspected that I probably could just delete the client-go directory. I personally think that client-go and staging should not be inside kubernetes. The script to synchronize it should be in client-go and not in kubernetes. This would remove a bug vector that took me 2.5 days to figure out what's going on. Thanks for your feedback.
With being merged we have removed on possible error of having another client-go in the GOPATH. If I understand you right, you were hit by this. So in master this shouldn't happen anymore. Best regards, Stefan , Vladimir Vivien <:
awesome, good to know. I did noticed this morning when I built, client-go was not placed on the GOPATH anymore. Thanks.


Question: running in the node. In the future, we plan to automatically provision some node services. <ins> In general, node controller is responsible for updating the NodeReady condition of node status to ConditionUnknown when a node becomes unreachable (e.g. due to the node being down), and then later evicting all the pods from the node (using graceful termination) if the node continues to be unreachable. (The current timeouts for those are 40s and 5m, respectively.) It also allocates CIDR blocks to the new nodes. </ins> ### Self-Registration of Nodes When kubelet flag `--register-node` is true (the default), the kubelet will attempt to

The expression right now is: "Node controller has a sync loop that creates/deletes Nodes from Kubernetes based on all matching VM instances listed from the cloud provider." But actually node controller does not create nodes from cloudprovider. Please refer to This issue was logged for the question with answer given: So the document expression should remove "creates" from the sentence.
Can we close this now that was merged?
Sure, close this issue. Thanks.


Question: } return cloudprovider.Zone{ <del> FailureDomain: failureDomain, Region:        to.String(vm.Location), </del> <ins> FailureDomain: strings.ToLower(failureDomain), Region:        strings.ToLower(to.String(vm.Location)), </ins> }, nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider gets the availability zone from IMDS directly and the zone is usually in lower cases in most regions. But there're chances this is not true in some regions, hence customer's workloads depend on availability zone may not work. What you expected to happen: The availability zone is always in lower cases. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig cloud-provider /area provider/azure
/assign
Fixed by .


Question: podClient = f.PodClient() }) <del> ginkgo.It("will start an ephemeral container in an existing pod", func() { </del> <ins> // Release: 1.25 // Testname: Ephemeral Container Creation // Description: Adding an ephemeral container to pod.spec MUST result in the container running. framework.ConformanceIt("will start an ephemeral container in an existing pod", func() { </ins> ginkgo.By("creating a target pod") pod := podClient.CreateSync(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "ephemeral-containers-target-pod"},

Issue tracking the addition of Ephemeral Containers to the . Exercising ephemeral containers in the node conformance test is required for
/assign /sig node /priority important-longterm /triage accepted
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /assign /sig node /priority important/long-term /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-longterm


Question: if strings.Contains(cfg.Global.NetworkName, "/") { networkURL = cfg.Global.NetworkName } else { <del> networkURL = gceNetworkURL(apiEndpoint, projectID, networkName) </del> <ins> networkURL = gceNetworkURL(apiEndpoint, projectID, cfg.Global.NetworkName) </ins> } }

Causes failures in:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig network /area platform/gce platform/gke

/kind bug /sig storage
/assign
It also fails on debian GKE. However, they all pass on debian and GCI GCE. Error msg is mounting timed out. This leads me to suspect that there is some difference in networking for GKE vs GCE.
I do strongly suspect there is some networking issue on GKE. Many other tests are also failing with "connection timed out" errors. 1.7 GKE tests are looking healthy, so it seems like something is broken on master. /sig network
Fixed by /close


Question: // For example: "Key1=Val1,Key2=Val2,KeyNoVal1=,KeyNoVal2" const ServiceAnnotationLoadBalancerAdditionalTags = "service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags" <ins> // Event key when a volume is stuck on attaching state when being attached to a volume const volumeAttachmentStuck = "VolumeAttachmentStuck" // Indicates that a node has volumes stuck in attaching state and hence it is not fit for scheduling more pods const nodeWithImpairedVolumes = "NodeWithImpairedVolumes" </ins> const ( // volumeAttachmentConsecutiveErrorLimit is the number of consecutive errors we will ignore when waiting for a volume to attach/detach volumeAttachmentStatusConsecutiveErrorLimit = 10

On AWS when we call "attach" of a volume then we periodically we check (via API call) if volume has indeed been attached. We run the check in a loop with timeout of 21 minutes. AWS documents that - if a volume is stuck in "attaching" state for too long, the node where volume is being attached is in "dirty" state and must be rebooted to fix the problem. Until node is rebooted, more pods with volumes to same node is only going be stuck in same state (i.e "attaching"). We propose that - if attach operation times out waiting for volume in "attaching" state, we should simply taint the node so as it can not affect other pods and a admin can manually fix the node. /sig aws /sig storage cc


Question: Expect(err).NotTo(HaveOccurred()) oldCond := client.ReplicaSetHasDesiredReplicas(c.Extensions(), oldRs) <del> wait.PollImmediate(10*time.Millisecond, 1*time.Minute, oldCond) </del> <ins> err = wait.PollImmediate(10*time.Millisecond, 1*time.Minute, oldCond) Expect(err).NotTo(HaveOccurred()) </ins> newCond := client.ReplicaSetHasDesiredReplicas(c.Extensions(), newRs) <del> wait.PollImmediate(10*time.Millisecond, 1*time.Minute, newCond) </del> <ins> err = wait.PollImmediate(10*time.Millisecond, 1*time.Minute, newCond) Expect(err).NotTo(HaveOccurred()) </ins> By(fmt.Sprintf("Updating the size (down) and template at the same time for deployment %q", deploymentName)) newReplicas = int32(5)

https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Happened on a presubmit run in . Previous issues for this test:
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s-

Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes

e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}


Question: } }(fsWatcher) <del> // Traverse plugin dir after starting the plugin processing goroutine if err := w.traversePluginDir(w.path); err != nil { w.Stop() return fmt.Errorf("failed to traverse plugin socket path %q, err: %v", w.path, err) } // Traverse deprecated plugin dir, if specified. if len(w.deprecatedPath) != 0 { if err := w.traversePluginDir(w.deprecatedPath); err != nil { w.Stop() return fmt.Errorf("failed to traverse deprecated plugin socket path %q, err: %v", w.deprecatedPath, err) } } </del> return nil }

What happened: A race condition in the plugin watcher might happen when a is created, started and immediately stopped and cleaned up. If executed several times, the unit test will expose this issue. The PR addressed part of the problem by serializing the handling of CREATE and DELETE events. However, a race might still happen in the function. describes in detail the problem. How to reproduce it (as minimally and precisely as possible): /sig storage /sig node
/assign


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/util/diff" </ins> "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/wait" utilfeature "k8s.io/apiserver/pkg/util/feature"

bug (): Originally posted by in cc /sig network /priority critical-urgent
fix in
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: t.Fatal(err) } LastObservedTime := metav1.MicroTime{Time: time.Now().Add(-9 * time.Minute)} createEvent := make(chan *v1beta1.Event, 10) updateEvent := make(chan *v1beta1.Event, 10) patchEvent := make(chan *v1beta1.Event, 10) <del> testEvents := testEventSeriesSink{ OnCreate: func(event *v1beta1.Event) (*v1beta1.Event, error) { createEvent <- event return event, nil }, OnUpdate: func(event *v1beta1.Event) (*v1beta1.Event, error) { updateEvent <- event return event, nil </del> <ins> table := []struct { patchFunc func(event *v1beta1.Event, patch []byte) (*v1beta1.Event, error) }{ { patchFunc: func(event *v1beta1.Event, patch []byte) (*v1beta1.Event, error) { // event we receive is already patched, usually the sink uses it //only to retrieve the name and namespace, here we'll use it directly. patchEvent <- event return event, nil }, </ins> }, <del> OnPatch: func(event *v1beta1.Event, patch []byte) (*v1beta1.Event, error) { // event we receive is already patched, usually the sink uses it //only to retrieve the name and namespace, here we'll use it directly. patchEvent <- event return event, nil </del> <ins> { patchFunc: func(event *v1beta1.Event, patch []byte) (*v1beta1.Event, error) { // we simulate an apiserver error here patchEvent <- nil return nil, &restclient.RequestConstructionError{} }, </ins> }, } <del> cache := map[eventKey]*v1beta1.Event{} eventBroadcaster := newBroadcaster(&testEvents, 0, cache).(*eventBroadcasterImpl) recorder := eventBroadcaster.NewRecorder(scheme.Scheme, "k8s.io/kube-foo").(*recorderImpl) cachedEvent := recorder.makeEvent(regarding, related, metav1.MicroTime{time.Now()}, v1.EventTypeNormal, "test", "some verbose message: 1", "eventTest", "eventTest-"+hostname, "started") cachedEvent.Series = &v1beta1.EventSeries{ Count:            10, LastObservedTime: LastObservedTime, } cacheKey := getKey(cachedEvent) cache[cacheKey] = cachedEvent eventBroadcaster.refreshExistingEventSeries() select { case <-patchEvent: t.Logf("validating event affected by patch request") eventBroadcaster.mu.Lock() defer eventBroadcaster.mu.Unlock() if len(cache) != 1 { t.Errorf("cache should be with same size, but instead got a size of %v", len(cache)) </del> <ins> for _, item := range table { testEvents := testEventSeriesSink{ OnCreate: func(event *v1beta1.Event) (*v1beta1.Event, error) { createEvent <- event return event, nil }, OnUpdate: func(event *v1beta1.Event) (*v1beta1.Event, error) { updateEvent <- event return event, nil }, OnPatch: item.patchFunc, </ins> } <del> // check that we emitted only one event if len(patchEvent) != 0 || len(createEvent) != 0 || len(updateEvent) != 0 { t.Errorf("exactly one event should be emitted, but got %v", len(patchEvent)) </del> <ins> cache := map[eventKey]*v1beta1.Event{} eventBroadcaster := newBroadcaster(&testEvents, 0, cache).(*eventBroadcasterImpl) recorder := eventBroadcaster.NewRecorder(scheme.Scheme, "k8s.io/kube-foo").(*recorderImpl) cachedEvent := recorder.makeEvent(regarding, related, metav1.MicroTime{time.Now()}, v1.EventTypeNormal, "test", "some verbose message: 1", "eventTest", "eventTest-"+hostname, "started") cachedEvent.Series = &v1beta1.EventSeries{ Count:            10, LastObservedTime: LastObservedTime, } cacheKey := getKey(cachedEvent) cache[cacheKey] = cachedEvent eventBroadcaster.refreshExistingEventSeries() select { case <-patchEvent: t.Logf("validating event affected by patch request") eventBroadcaster.mu.Lock() defer eventBroadcaster.mu.Unlock() if len(cache) != 1 { t.Errorf("cache should be with same size, but instead got a size of %v", len(cache)) } // check that we emitted only one event if len(patchEvent) != 0 || len(createEvent) != 0 || len(updateEvent) != 0 { t.Errorf("exactly one event should be emitted, but got %v", len(patchEvent)) } cacheEvent, exists := cache[cacheKey] if cacheEvent == nil || !exists { t.Errorf("expected event to exist and not being nil, but instead event: %v and exists: %v", cacheEvent, exists) } case <-time.After(wait.ForeverTestTimeout): t.Fatalf("timeout after %v", wait.ForeverTestTimeout) </ins> } <del> case <-time.After(wait.ForeverTestTimeout): t.Fatalf("timeout after %v", wait.ForeverTestTimeout) </del> } }

What happened: kube-scheduler panics when client-go tries to finish the existing event series, see: What you expected to happen: kube-scheduler should not panic How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority critical-urgent /sig scheduling


Question: newURL.Path = path.Join(h.canonicalPrefix, req.URL.Path) newURL.RawQuery = "" newURL.Fragment = "" <del> return h.selfLinker.SetSelfLink(obj, newURL.String()) </del> <ins> err := h.selfLinker.SetSelfLink(obj, newURL.String()) if err != nil { return err } if !runtime.IsListType(obj) { return nil } // Set self-link of objects in the list. items, err := runtime.ExtractList(obj) if err != nil { return err } for i := range items { if err := h.setSelfLinkAddName(items[i], req); err != nil { return err } } return runtime.SetList(obj, items) </ins> } // Like setSelfLink, but appends the object's name.

Had a report of this over email. Filing this issue so we don't forget to investigate.
Can't reproduce.


Question: }, expectedTrace: []*spanExpectation{ { <del> name: "KubernetesAPI", </del> <ins> name: "PUT /api/v1/nodes/{:name}", </ins> attributes: map[string]func(*commonv1.AnyValue) bool{ "http.user_agent": func(v *commonv1.AnyValue) bool { return strings.HasPrefix(v.GetStringValue(), "tracing.test")

When enable apiserver tracing, the webhook request spans only show spanName as /, mostly because use http request operations as default span name, which is hard to understand. ! Following the open-telemetry standard of spanName, it's recommended to use as spanName. ! SpanName of webhook request should be , like ! APIServerTracing feature gates of apiserver a pod tracing dashboard No response <details18 </details<detailsNone </details<details</details<details</details<details</details<details</details
I make a fix for this.
/sig api-machinery instrumentation
Do you know if this is fixed in later versions of opentelemetry?
/triage accepted /assign
Looks like might be related
not yet, I think we can fix this in otel upstream, but it's hard to backport to otel , which is the version of otel used in Kubernetes
Good point. It think fixing it here is a good idea in that case.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
could I take on this issue?
yes. Feel free to work on this.
! I test locally by 1.29 kubernetes and start a sample webhook, looks like the client span has already conforms to semantic conventions , which should be in the form of "HTTP {METHODNAME}"
! For apiserver as a server, now the spanName is still KubernetesAPI, which I think we could enhance it into "HTTP get_account/{accountId}" format
After digging for a while, I found one solution to implement this, I draft this idea in , could you help review about whether this is a correct way to implement it?


Question: } } <del> // // TestUnreserveReservePlugin tests invocation of the Unreserve operation in // // reserve plugins through failures in execution points such as pre-bind. Also // // tests that the order of invocation of Unreserve operation is executed in the // // reverse order of invocation of the Reserve operation. // func TestReservePluginUnreserve(t *testing.T) { // 	tests := []struct { // 		name             string // 		failReserve      bool // 		failReserveIndex int // 		failPreBind      bool // 	}{ // 		{ // 			name:             "fail reserve", // 			failReserve:      true, // 			failReserveIndex: 1, // 		}, // 		{ // 			name:        "fail preBind", // 			failPreBind: true, // 		}, // 		{ // 			name: "pass everything", // 		}, // 	} // 	for _, test := range tests { // 		t.Run(test.name, func(t *testing.T) { // 			numReservePlugins := 3 // 			pluginInvokeEventChan := make(chan pluginInvokeEvent, numReservePlugins) // 			preBindPlugin := &PreBindPlugin{ // 				failPreBind: true, // 			} // 			var reservePlugins []*ReservePlugin // 			for i := 0; i < numReservePlugins; i++ { // 				reservePlugins = append(reservePlugins, &ReservePlugin{ // 					name:                  fmt.Sprintf("%s-%d", reservePluginName, i), // 					pluginInvokeEventChan: pluginInvokeEventChan, // 				}) // 			} // 			registry := frameworkruntime.Registry{ // 				// TODO(#92229): test more failure points that would trigger Unreserve in // 				// reserve plugins than just one pre-bind plugin. // 				preBindPluginName: newPlugin(preBindPlugin), // 			} // 			for _, pl := range reservePlugins { // 				registry[pl.Name()] = newPlugin(pl) // 			} // 			// Setup initial reserve and prebind plugin for testing. // 			prof := schedulerconfig.KubeSchedulerProfile{ // 				SchedulerName: v1.DefaultSchedulerName, // 				Plugins: &schedulerconfig.Plugins{ // 					Reserve: schedulerconfig.PluginSet{ // 						// filled by looping over reservePlugins // 					}, // 					PreBind: schedulerconfig.PluginSet{ // 						Enabled: []schedulerconfig.Plugin{ // 							{ // 								Name: preBindPluginName, // 							}, // 						}, // 					}, // 				}, // 			} // 			for _, pl := range reservePlugins { // 				prof.Plugins.Reserve.Enabled = append(prof.Plugins.Reserve.Enabled, schedulerconfig.Plugin{ // 					Name: pl.Name(), // 				}) // 			} // 			// Create the master and the scheduler with the test plugin set. // 			testCtx := initTestSchedulerForFrameworkTest(t, testutils.InitTestMaster(t, "reserve-plugin-unreserve", nil), 2, // 				scheduler.WithProfiles(prof), // 				scheduler.WithFrameworkOutOfTreeRegistry(registry)) // 			defer testutils.CleanupTest(t, testCtx) // 			preBindPlugin.failPreBind = test.failPreBind // 			if test.failReserve { // 				reservePlugins[test.failReserveIndex].failReserve = true // 			} // 			// Create a best effort pod. // 			pod, err := createPausePod(testCtx.ClientSet, // 				initPausePod(&pausePodConfig{Name: "test-pod", Namespace: testCtx.NS.Name})) // 			if err != nil { // 				t.Errorf("Error while creating a test pod: %v", err) // 			} // 			if test.failPreBind || test.failReserve { // 				if err = wait.Poll(10*time.Millisecond, 30*time.Second, podSchedulingError(testCtx.ClientSet, pod.Namespace, pod.Name)); err != nil { // 					t.Errorf("Expected a scheduling error, but didn't get it: %v", err) // 				} // 				for i := numReservePlugins - 1; i >= 0; i-- { // 					select { // 					case event := <-pluginInvokeEventChan: // 						expectedPluginName := reservePlugins[i].Name() // 						if expectedPluginName != event.pluginName { // 							t.Errorf("event.pluginName = %s, want %s", event.pluginName, expectedPluginName) // 						} // 					case <-time.After(time.Second * 30): // 						t.Errorf("pluginInvokeEventChan receive timed out") // 					} // 				} // 			} else { // 				if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, pod); err != nil { // 					t.Errorf("Expected the pod to be scheduled, got an error: %v", err) // 				} // 				for i, pl := range reservePlugins { // 					if pl.numUnreserveCalled != 0 { // 						t.Errorf("reservePlugins[%d].numUnreserveCalled = %d, want 0", i, pl.numUnreserveCalled) // 					} // 				} // 			} // 			testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{pod}) // 		}) // 	} // } </del> <ins> // TestUnReserveReservePlugins tests invocation of the Unreserve operation in // reserve plugins through failures in execution points such as pre-bind. Also // tests that the order of invocation of Unreserve operation is executed in the // reverse order of invocation of the Reserve operation. func TestUnReserveReservePlugins(t *testing.T) { tests := []struct { name          string plugins       []*ReservePlugin fail          bool failPluginIdx int }{ { name:          "The first Reserve plugin fails", failPluginIdx: 0, plugins: []*ReservePlugin{ { name:        "failedReservePlugin1", failReserve: true, }, { name:        "succeedReservePlugin", failReserve: false, }, { name:        "failedReservePlugin2", failReserve: true, }, }, fail: true, }, { name:          "The middle Reserve plugin fails", failPluginIdx: 1, plugins: []*ReservePlugin{ { name:        "succeedReservePlugin1", failReserve: false, }, { name:        "failedReservePlugin1", failReserve: true, }, { name:        "succeedReservePlugin2", failReserve: false, }, }, fail: true, }, { name:          "The last Reserve plugin fails", failPluginIdx: 2, plugins: []*ReservePlugin{ { name:        "succeedReservePlugin1", failReserve: false, }, { name:        "succeedReservePlugin2", failReserve: false, }, { name:        "failedReservePlugin1", failReserve: true, }, }, fail: true, }, { name:          "The Reserve plugins succeed", failPluginIdx: -1, plugins: []*ReservePlugin{ { name:        "succeedReservePlugin1", failReserve: false, }, { name:        "succeedReservePlugin2", failReserve: false, }, { name:        "succeedReservePlugin3", failReserve: false, }, }, fail: false, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { var pls []framework.Plugin for _, pl := range test.plugins { pls = append(pls, pl) } registry, prof := initRegistryAndConfig(t, pls...) // Create the API server and the scheduler with the test plugin set. testCtx := initTestSchedulerForFrameworkTest( t, testutils.InitTestAPIServer(t, "unreserve-reserve-plugin", nil), 2, scheduler.WithProfiles(prof), scheduler.WithFrameworkOutOfTreeRegistry(registry)) defer testutils.CleanupTest(t, testCtx) // Create a best effort pod. podName := "test-pod" pod, err := createPausePod(testCtx.ClientSet, initPausePod(&pausePodConfig{Name: podName, Namespace: testCtx.NS.Name})) if err != nil { t.Errorf("Error while creating a test pod: %v", err) } if test.fail { if err = wait.Poll(10*time.Millisecond, 30*time.Second, podSchedulingError(testCtx.ClientSet, pod.Namespace, pod.Name)); err != nil { t.Errorf("Expected a reasons other than Unschedulable, but got: %v", err) } for i, pl := range test.plugins { if i <= test.failPluginIdx { if pl.numReserveCalled != 1 { t.Errorf("Reserve Plugins %s numReserveCalled = %d, want 1.", pl.name, pl.numReserveCalled) } } else { if pl.numReserveCalled != 0 { t.Errorf("Reserve Plugins %s numReserveCalled = %d, want 0.", pl.name, pl.numReserveCalled) } } if pl.numUnreserveCalled != 1 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 1.", pl.name, pl.numUnreserveCalled) } } } else { if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, pod); err != nil { t.Errorf("Expected the pod to be scheduled. error: %v", err) } for _, pl := range test.plugins { if pl.numReserveCalled != 1 { t.Errorf("Reserve Plugin %s numReserveCalled = %d, want 1.", pl.name, pl.numReserveCalled) } if pl.numUnreserveCalled != 0 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 0.", pl.name, pl.numUnreserveCalled) } } } testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{pod}) }) } } // TestUnReservePermitPlugins tests unreserve of Permit plugins. func TestUnReservePermitPlugins(t *testing.T) { tests := []struct { name   string plugin *PermitPlugin fail   bool }{ { name: "All Reserve plugins passed, but a Permit plugin was rejected", fail: true, plugin: &PermitPlugin{ name:         "rejectedPermitPlugin", rejectPermit: true, }, }, { name: "All Reserve plugins passed, but a Permit plugin timeout in waiting", fail: true, plugin: &PermitPlugin{ name:          "timeoutPermitPlugin", timeoutPermit: true, }, }, { name: "The Permit plugin succeed", fail: false, plugin: &PermitPlugin{ name: "succeedPermitPlugin", }, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { reservePlugin := &ReservePlugin{ name:        "reservePlugin", failReserve: false, } registry, profile := initRegistryAndConfig(t, []framework.Plugin{test.plugin, reservePlugin}...) // Create the API server and the scheduler with the test plugin set. testCtx := initTestSchedulerForFrameworkTest( t, testutils.InitTestAPIServer(t, "unreserve-reserve-plugin", nil), 2, scheduler.WithProfiles(profile), scheduler.WithFrameworkOutOfTreeRegistry(registry)) defer testutils.CleanupTest(t, testCtx) // Create a best effort pod. podName := "test-pod" pod, err := createPausePod(testCtx.ClientSet, initPausePod(&pausePodConfig{Name: podName, Namespace: testCtx.NS.Name})) if err != nil { t.Errorf("Error while creating a test pod: %v", err) } if test.fail { if err = waitForPodUnschedulable(testCtx.ClientSet, pod); err != nil { t.Errorf("Didn't expect the pod to be scheduled. error: %v", err) } // Verify the Reserve Plugins if reservePlugin.numUnreserveCalled != 1 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 1.", reservePlugin.name, reservePlugin.numUnreserveCalled) } } else { if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, pod); err != nil { t.Errorf("Expected the pod to be scheduled. error: %v", err) } // Verify the Reserve Plugins if reservePlugin.numUnreserveCalled != 0 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 0.", reservePlugin.name, reservePlugin.numUnreserveCalled) } } if test.plugin.numPermitCalled != 1 { t.Errorf("Expected the Permit plugin to be called.") } testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{pod}) }) } } // TestUnReservePreBindPlugins tests unreserve of Prebind plugins. func TestUnReservePreBindPlugins(t *testing.T) { tests := []struct { name   string plugin *PreBindPlugin fail   bool }{ { name: "All Reserve plugins passed, but a PreBind plugin failed", fail: true, plugin: &PreBindPlugin{ podUIDs:       make(map[types.UID]struct{}), rejectPreBind: true, }, }, { name:   "All Reserve plugins passed, and PreBind plugin succeed", fail:   false, plugin: &PreBindPlugin{podUIDs: make(map[types.UID]struct{})}, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { reservePlugin := &ReservePlugin{ name:        "reservePlugin", failReserve: false, } registry, profile := initRegistryAndConfig(t, []framework.Plugin{test.plugin, reservePlugin}...) // Create the API server and the scheduler with the test plugin set. testCtx := initTestSchedulerForFrameworkTest( t, testutils.InitTestAPIServer(t, "unreserve-prebind-plugin", nil), 2, scheduler.WithProfiles(profile), scheduler.WithFrameworkOutOfTreeRegistry(registry)) defer testutils.CleanupTest(t, testCtx) // Create a pause pod. podName := "test-pod" pod, err := createPausePod(testCtx.ClientSet, initPausePod(&pausePodConfig{Name: podName, Namespace: testCtx.NS.Name})) if err != nil { t.Errorf("Error while creating a test pod: %v", err) } if test.fail { if err = wait.Poll(10*time.Millisecond, 30*time.Second, podSchedulingError(testCtx.ClientSet, pod.Namespace, pod.Name)); err != nil { t.Errorf("Expected a reasons other than Unschedulable, but got: %v", err) } // Verify the Reserve Plugins if reservePlugin.numUnreserveCalled != 1 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 1.", reservePlugin.name, reservePlugin.numUnreserveCalled) } } else { if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, pod); err != nil { t.Errorf("Expected the pod to be scheduled. error: %v", err) } // Verify the Reserve Plugins if reservePlugin.numUnreserveCalled != 0 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 0.", reservePlugin.name, reservePlugin.numUnreserveCalled) } } if test.plugin.numPreBindCalled != 1 { t.Errorf("Expected the Prebind plugin to be called.") } testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{pod}) }) } } // TestUnReserveBindPlugins tests unreserve of Bind plugins. func TestUnReserveBindPlugins(t *testing.T) { tests := []struct { name   string plugin *BindPlugin fail   bool }{ { name:   "All Reserve plugins passed, and Bind plugin succeed", fail:   false, plugin: &BindPlugin{name: "SucceedBindPlugin"}, }, { name:   "All Reserve plugins passed, but a Bind plugin failed", fail:   false, plugin: &BindPlugin{name: "FailedBindPlugin"}, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { reservePlugin := &ReservePlugin{ name:        "reservePlugin", failReserve: false, } registry, profile := initRegistryAndConfig(t, []framework.Plugin{test.plugin, reservePlugin}...) apiCtx := testutils.InitTestAPIServer(t, "unreserve-bind-plugin", nil) test.plugin.client = apiCtx.ClientSet // Create the scheduler with the test plugin set. testCtx := initTestSchedulerForFrameworkTest( t, apiCtx, 2, scheduler.WithProfiles(profile), scheduler.WithFrameworkOutOfTreeRegistry(registry)) defer testutils.CleanupTest(t, testCtx) // Create a pause pod. podName := "test-pod" pod, err := createPausePod(testCtx.ClientSet, initPausePod(&pausePodConfig{Name: podName, Namespace: testCtx.NS.Name})) if err != nil { t.Errorf("Error while creating a test pod: %v", err) } if test.fail { if err = wait.Poll(10*time.Millisecond, 30*time.Second, podSchedulingError(testCtx.ClientSet, pod.Namespace, pod.Name)); err != nil { t.Errorf("Expected a reasons other than Unschedulable, but got: %v", err) } // Verify the Reserve Plugins if reservePlugin.numUnreserveCalled != 1 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 1.", reservePlugin.name, reservePlugin.numUnreserveCalled) } } else { if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, pod); err != nil { t.Errorf("Expected the pod to be scheduled. error: %v", err) } // Verify the Reserve Plugins if reservePlugin.numUnreserveCalled != 0 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 0.", reservePlugin.name, reservePlugin.numUnreserveCalled) } } if test.plugin.numBindCalled != 1 { t.Errorf("Expected the Prebind plugin to be called.") } testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{pod}) }) } } </ins> type pluginInvokeEvent struct { pluginName string

See - currently there's just one test that triggers Unreserve through a pre-bind failure.
/sig scheduling
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I shall work on it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/assign
i saw commented the only test code, do you have any idea why this going on?
well, I found some functions missing, maybe because of the code refactor.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale