Question: ginkgo.By("Run a high priority pod that has same requirements as that of lower priority pod") // Create a high priority pod and make sure it is scheduled on the same node as the low priority pod. <del> runPausePod(f, pausePodConfig{ </del> <ins> runPausePodWithTimeout(f, pausePodConfig{ </ins> Name:              "preemptor-pod", PriorityClassName: highPriorityClassName, Resources: &v1.ResourceRequirements{ Requests: podRes, Limits:   podRes, }, <del> }) </del> <ins> }, framework.PodStartShortTimeout) </ins> preemptedPod, err := cs.CoreV1().Pods(pods[0].Namespace).Get(context.TODO(), pods[0].Name, metav1.GetOptions{}) podPreempted := (err != nil && apierrors.IsNotFound(err)) ||

Which jobs are flaking: sig-release-master-blocking Conformance - GCE - master Which test(s) are flaking: E2E: [sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance] Testgrid link: Reason for failure: Anything else we need to know: is currently unavailable links to specific failures in spyglass appreciated From TestGrid alert ...
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
I cannot spot any significant logic flaw in the test. A possible cause is the timeout is set too short (1 minute), which may be not enough for the terminating pod to run. Notice that unlike integration test, the time includes the end to end cycle: (1) preemptor being scheduled and evict low-priority pods, (2) low-priority pods being terminating -reported by kubelet that the victims have been deleted, (3) scheduler gets notifications from API server that the victims have been deleted, (4) preemptor gets scheduled, (5) kubelet reports the preemptor is running. /cc for more thoughts.
Maybe, some logs to / would help. Just making the timeout longer don't make sense.
Is addressing this flake?
I think this flake happens before the failure tries to fix:
/kind flake
That's indeed a lot of steps, and flakiness seems very low. I think increasing the timeout for preemption seems reasonable.
is this flake a must-fix in 1.20?


Question: metric   *metricInfo podReadiness []v1.ConditionStatus <ins> podPhase     []v1.PodPhase </ins> } const (

/kind bug What happened: HPA fails to launch additional pods due to pods in evicted state, even with utilization significantly over target. What you expected to happen: When HPA pods reach the target utilization, new pods should be allocated and provisioned. How to reproduce it (as minimally and precisely as possible): Launch a HPA, let the pods fill the disk on the host which will trigger evictions due to "Pod The node was low on resource: [DiskPressure]", HPA will exceed target and not allocate additional pods. Anything else we need to know?: Seems related to and though those are not specific to HPA. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): cos-stable-57-9202-64-0 kernel commit id Kernel (e.g. ):Linux gke-prod1-use1b-v2--35s4 4.4.70+ SMP Mon Sep 18 12:43:08 PDT 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux
/sig scheduling
Are there any workarounds while we wait for a fix? We have ran into this on a 1.8.4 GKE cluster.
/sig autoscaling /remove-sig scheduling
I am thinking of fixing this by excluding pods with Phase=Failed from the unreadyPods that are taken into account. It seems the right thing to do as they will not become running again. Do you see any issues?
that seems decent -- anything that will never be running again should not be factored into our calculations.
/status approved-for-milestone Adding to milestone -- if the HPA isn't functioning in the case of evictions, that seems like a decently severe HPA bug.
oops, missing priority was my fault, moving back into milestone: /priority critical-urgent /remove-priority important-soon
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues.

Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: go hpaController.Run(stop) tc.Lock() <del> if tc.verifyEvents { tc.Unlock() </del> <ins> shouldWait := tc.verifyEvents tc.Unlock() if shouldWait { </ins> // We need to wait for events to be broadcasted (sleep for longer than record.sleepDuration). <del> time.Sleep(2 * time.Second) </del> <ins> timeoutTime := time.Now().Add(2 * time.Second) for now := time.Now(); timeoutTime.After(now); now = time.Now() { sleepUntil := timeoutTime.Sub(now) select { case <-tc.processed: // drain the chan of any sent events to keep it from filling before the timeout case <-time.After(sleepUntil): // timeout reached, ready to verifyResults } } </ins> } else { <del> tc.Unlock() </del> <ins> // Wait for HPA to be processed. <-tc.processed </ins> } <del> // Wait for HPA to be processed. <-tc.processed </del> tc.verifyResults(t) }

What happened: The unit test for Horizontal Pod Autoscaler frequently deadlocked on the OpenShift CI system. What you expected to happen: The unit test should be robust and succeed without having to win a race condition to avoid deadlock. How to reproduce it (as minimally and precisely as possible): I'm not sure. Anything else we need to know?: Here are some logs of the problem happening: https://openshift-gce- The deadlocks seem to happen when the chan fills up while waiting for events. If that happens, there's a race where immediately after an event is read but before it can acquire the lock to verify the test, 2 event reactors are able to acquire the lock and write to the chan. The second will block forever since the chan is full and there's nobody to read from it. I've opened a PR with the fix. See
/sig autoscaling
/assign


Question: return nil } <ins> // TODO turn this into reusable method checking available resources func contains(resourcesList map[string]*unversioned.APIResourceList, resource unversioned.GroupVersionResource) bool { if resourcesList == nil { return false } resourcesGroup, ok := resourcesList[resource.GroupVersion().String()] if !ok { return false } for _, item := range resourcesGroup.APIResources { if resource.Resource == item.Name { return true } } return false } </ins> func waitForPodRunning(c *client.Client, pod *api.Pod, out io.Writer) (status api.PodPhase, err error) { for { pod, err := c.Pods(pod.Namespace).Get(pod.Name)

Tested with: Most of the simple basic starting guides direct users to create a cluster with . If I run kubectl 1.2 against a 1.1 cluster, I'll get the following behavior now I happen to know I can get the old behavior by doing but I don't expect an average user to know that or be able to find it easily. I'm mainly concerned that we will be breaking recommended flow for GKE users if they happen to use a new kubectl before their cluster's master is upgraded. We can just update our docs, but breaking a "porcelain" command seems like poor form. Other options include defaulting to the generator for a few patch releases, and/or making do some kind of version check so it can fallback to the generator if deployments/jobs are not available. thoughts?
We should update those docs and document how to achieve stable behavior: .
This would ideally use API discovery, though, to check whether the API exists.
How hard would it be to check whether Deployment existed? We had the same issue in 1.1, btw, with Job.
I'd accept a hack, also, which checked the k8s version if there wasn't enough discovery info in 1.1.
We have the same issue with Job in 1.2, also.
Fix is in


Question: } if sgNeedsUpdate { glog.V(3).Infof("ensure(%s): sg(%s) - updating", serviceName, *sg.Name) <ins> // azure-sdk-for-go introduced contraint validation which breaks the updating here if we don't set these // to nil. This is a workaround until https://github.com/Azure/go-autorest/issues/112 is fixed sg.SecurityGroupPropertiesFormat.NetworkInterfaces = nil sg.SecurityGroupPropertiesFormat.Subnets = nil </ins> _, err := az.SecurityGroupsClient.CreateOrUpdate(az.ResourceGroup, *sg.Name, sg, nil) if err != nil { return nil, err

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure / Azure Container Service OS (e.g. from /etc/os-release): Xenial Kernel (e.g. ): latest lts Install tools: kubelet + static manifests Others: n/a What happened: I created a Service (type=LoadBalancer). The load balancer did not complete provisioning and security group was not updated, so the LB that is there doesn't work. What you expected to happen: Azure load balancer is created, external IP field is populated (doesn't show as "Pending") How to reproduce it (as minimally and precisely as possible): Try it. Anything else do we need to know: No. There is an issue opened against the repo that caused this problem: This was reported here: CC:
Should we do a fix that simply sets the 2 properties to nil before a real fix comes into go-autorest?
I'd prefer to fix it properly in but as of yet, it's unclear what the timeframe is. Let me ask for a timeline.
I agree that a proper fix in go-autorest would be preferable. But as long as it's not fixed, LBs are broken on k8s=1.5.2. That's why I would like to do a workaround until the real fix comes in.
Is there a workaround? Is there a way I can manually intervene to make the LoadBalancer work?
No. Not currently, other than building your own copy of with patch applied. I'm testing a more proper fix today or tomorrow - hope to have it in for 1.5.3.


Question: defaultMetricsHandler(w, req) }) } <ins> // register apiserver and etcd metrics func register() { apimetrics.Register() etcdmetrics.Register() } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: endpoint of different k8s components expose Prometheus metrics that are not for them. For example, exposes metrics: or : What you expected to happen: Prometheus metrics only for to appear on kubelet's endpoint. How to reproduce it (as minimally and precisely as possible): Access the endpoint of all Kubernetes components Anything else we need to know?: This problem comes from usage of functions in different packages that automatically register Prometheus metrics. Just importing that package or some package that depends on it can cause those metrics to appear in unexpected places. In the list bellow you can find the different packages that have init functions and affected binaries: [ ] () cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] () cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubectl kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet [ ] hyperkube kube-apiserver kube-controller-manager [ ] hyperkube kube-apiserver [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet Environment: Kubernetes version (use ):Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A /sig instrumentation
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err != nil { return KeyError{obj, err} } <del> newDeltas := append(f.items[id], Delta{actionType, obj}) </del> <ins> oldDeltas := f.items[id] newDeltas := append(oldDeltas, Delta{actionType, obj}) </ins> newDeltas = dedupDeltas(newDeltas) if len(newDeltas) > 0 {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In reviewing another PR, I noticed incorrect commentary and code. The primary offense is at , where a comment asserts (incorrectly) that a key can be present in but not and the code removes a key from without ensuring it is absent from . The only saving grace is that this code is actually unreachable (as also noted in the comment). I think it would be better to fix the comment and in this unreachable situation, than to violate an invariant that is critical to other code in this file. also contemplates the possibility that a key can be in and absent from . What you expected to happen: I expected there to be no comments or code that contemplate violations of the invariant at How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): current branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Sorry, I should have noticed this in my earlier pass over this code.
Agreed. Was discussing this with at
We discussed this one, Mike will send PR, Daniel will review.


Question: component: kube-proxy spec: hostNetwork: true <ins> initContainers: - name: touch-lock image: busybox command: ['/bin/touch', '/run/xtables.lock'] securityContext: privileged: true volumeMounts: - mountPath: /run name: run readOnly: false </ins> containers: - name: kube-proxy image: {{pillar['kube_docker_registry']}}/kube-proxy:{{pillar['kube-proxy_docker_tag']}}

BUG REPORT Kubernetes version 2 Environment: Install tools: What happened: The code in takes care to supply the flag so it will wait on a lock to avoid interference between multiple processes modifying the table. If you read the source you discover it does this by creating a file . However, when is running in a container, is in a local filesystem, so it is not able to lock against processes running on the host or in other containers. And before you decide that the answer is to mount from the host, look at all the issues around leaking netns, e.g. Credit for pointing this out to me. Credit for pointing out locking functionality was to a few months back
Ironically, this is caused by a bugfix that moved to file-based locking. Previously, iptables would lock by reserving an abstract unix socket, which is tied to the network namespace.
Some discussion also in and following comments.
I experimented with this: but it didn't work on CoreOS, as the host wasn't using a version of iptables that used the file lock yet. Just ended up creating a directory call
I guess I should file this as an issue against kubeadm and any of the other sources of kube-proxy container config.
/assign
Should we leave this open (waiting for ... ?)
Waiting for I guess... (which I'm reviewing now)
i think this should not be marked as complete until the kubeadm fixes are in.
/reopen
/close fix in


Question: google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= <del> gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= k8s.io/klog/v2 v2.100.1 h1:7WCHKK6K8fNhTqfBhISHQ97KrnJNFZMcQvKp7gP/tmg= k8s.io/klog/v2 v2.100.1/go.mod h1:y1WjHnz7Dj687irZUWR/WLkLc5N1YHtjLdmgWjndZn0= k8s.io/kube-openapi v0.0.0-20231010175941-2dd684a91f00/go.mod h1:AsvuZPBlUDVuCdzJ87iajxtXuR9oktsTctW/R9wwouA= k8s.io/utils v0.0.0-20230726121419-3b25d923346b h1:sgn3ZU783SCgtaSJjpcVVlRqd6GSnlTLKgpAAttJvpI= k8s.io/utils v0.0.0-20230726121419-3b25d923346b/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0= sigs.k8s.io/structured-merge-diff/v4 v4.3.0/go.mod h1:N8hJocpFajUSSeSJ9bOZ77VzejKZaXsTtZo4/u7Io08= sigs.k8s.io/yaml v1.3.0/go.mod h1:GeOyir5tyXNByN85N/dRIT9es5UQNerPYEKK56eTBm8= </del>

Drop the and dependencies from to keep the import lighter. /sig auth /triage accepted
/assign Hi, , I created a PR to fix this. I am not sure If I am correct due to the repo seems generated. Would you like to provides some hints for me?
This will likely require a new go module in to move the client-go dep and then mark it as internal, ex: . is the only API machinery dep so we can probably come up with some way to re-implement the wait logic locally? Then we drop these lines so that these imports do not come back again:
Whoa, yeah, I'd take a for loop and a time.Sleep in a test to drop those deps
What is kms using client-go for? Tests only?
These were all needed by the KMS repo back when we had a much more complicated reference implementation which required a key hierarchy. Now they are just used in some tests and mock implementations, so we should be able to simplify things.
I found a place for rate limit. #liggitt
that package just delegates to "", switch that mock provider to use that directly?


Question: return []unversioned.GroupVersion{} } <del> return append([]unversioned.GroupVersion{}, groupMeta.GroupVersions...) </del> <ins> ret := []unversioned.GroupVersion{} for _, version := range groupMeta.GroupVersions { if m.IsEnabledVersion(version) { ret = append(ret, version) } } return ret </ins> } // Group returns the metadata of a group if the group is registered, otherwise

introduced a bug where doesn't consult anymore... I think. I was trying to debug another problem and spotted this. I think only clients use this function. APIServer has an entirely different mechanism for tracking enabled/disabled things. cc


Question: To start using your cluster, open up another terminal/tab and run: <del> export KUBERNETES_PROVIDER=local </del> <ins> cluster/kubectl.sh config set-cluster local --server=http://${API_HOST}:${API_PORT} --insecure-skip-tls-verify=true --global cluster/kubectl.sh config set-context local --cluster=local --global cluster/kubectl.sh config use-context local </ins> cluster/kubectl.sh EOF

I've been pulling my hair out trying get kubectl talk to my local cluster. It seems that kubectl merges the local file with values pulled down from the cloud. I've done the following: The I just set is silently overridden. Questions: is this documented anywhere? should kubectl merge in my GCE cluster when I set my cloud-provider to something else? should kubectl override my previously set current-context?
workaround is to use flag to override the config file value.
but not a good UX long term
Someone else hit this as well: . Better fix is to have the local-up script set a context, which I will try to add today. As for the overwriting behavior, it's documented in , but based on what you describe, the intended load order is not working; local .kubeconfig should overwrite global. cc
That sounds infuriating. I'll take a look this morning.


Question: // that we can register that the api-server is no longer ready while we attempt to gracefully // shutdown. func (s *GenericAPIServer) addReadyzShutdownCheck(stopCh <-chan struct{}) error { <del> return s.AddReadyzChecks(shutdownCheck{stopCh}) </del> <ins> return s.AddReadyzChecks(healthz.NewShutdownHealthz(stopCh)) </ins> } // installHealthz creates the healthz endpoint for this server

Quoted from 's Splitting healthz/ into livez/ and readyz/
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/sig scheduling
I read all the content of the PR comment and am willing to work on this issue Do we need to continue some discussion for this issue
Hi , thanks for offering your help. But I was working in progress before knowing you can work on this. However, please feel free to contribute the similar issue
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: return } <del> // we need to wrap the roundtripper in another roundtripper which will apply the front proxy headers proxyRoundTripper, upgrade, err := maybeWrapForConnectionUpgrades(handlingInfo.restConfig, handlingInfo.proxyRoundTripper, req) if err != nil { proxyError(w, req, err.Error(), http.StatusInternalServerError) return } </del> <ins> proxyRoundTripper := handlingInfo.proxyRoundTripper upgrade := httpstream.IsUpgradeRequest(req) </ins> proxyRoundTripper = transport.NewAuthProxyRoundTripper(user.GetName(), user.GetGroups(), user.GetExtra(), proxyRoundTripper) // if we are upgrading, then the upgrade path tries to use this request with the TLS config we provide, but it does

What happened: upgrade requests to aggregated APIs (like websocket-based watch requests) on a kube-apiserver using --ssh-user hang What you expected to happen: Websocket-based watch requests work properly How to reproduce it (as minimally and precisely as possible): a kube-apiserver configured to use a node dialer to reach services/pods (e.g. with ) an aggregated API server (like the sample-apiserver) on a node other than the kube-apiserver a websocket-based watch against the aggregated API Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery /cc lavalamp
The code in question is here: handlingInfo.proxyRoundTripper is built here: and inherits the proxyDialerFn built in from the following construction chain: However, drops the custom dialer contained inside and from the returned round tripper. That means that upgrade requests fall back to use the default dialer, which cannot reach the aggregated API server
cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/assign would be good to resolve with the apiserver proxy work


Question: v1 "k8s.io/api/core/v1" storage "k8s.io/api/storage/v1" v1beta1storage "k8s.io/api/storage/v1beta1" <del> apierrors "k8s.io/apimachinery/pkg/api/errors" </del> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: This switches NodeLister to scheduler cache for . However, the new implementation in the cache never returns IsNotFound error any more. Rather, it returns a normal error: However, we rely on the error type for logging, for example: What you expected to happen: The cache should return IsNotFound error if applicable. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig scheduling cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi. I'd like to work on this for #Hacktoberfest, and for my very first open-source contribution.
Hello, I would love to contribute too in case. I'm already looking at the code in fact :D
Thanks all for the interest, will assign to since he expressed interest first. /assign
I was browsing around the rest of the implementation of the cache and noticed that none of the other accessors return . Also, I couldn't determine if ever returned anything other than an . I'm new here, so I may be doing it wrong. Are you sure the behavior changed, and are you sure returning is expected? Should the other accessors be changed, too?
Let's look at previous release. The object is initialized in [] where it creates a .

Now let's take a look at implementation: There you can see that it calls NodeLister Get() method, which can return a IsNotFound error.
Hello, is this issue being solved ? If not, i am interested in solving this issue.
Thanks but I am working on this currently.
, if you haven't started your work yet, let me know if I can raise a PR for this issue. I have the change ready.
I started, but haven’t been able to finish creating the PR. Go ahead and submit yours. Thanks. Sent from my iPhone


Question: // CleanScope returns the scope of the request. func CleanScope(requestInfo *request.RequestInfo) string { <del> if requestInfo.Namespace != "" { return "namespace" } </del> if requestInfo.Name != "" { return "resource" } <ins> if requestInfo.Namespace != "" { return "namespace" } </ins> if requestInfo.IsResourceRequest { return "cluster" }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --the issue stays in the CleanScope function where all GET calls will end up with scope "namespace" and end up with verb "LIST" Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /sig api-machinery
/triage accepted


Question: responses: 200: body: <del> application/json: example: !include examples/pod.json </del> <ins> example: !include examples/pod.json </ins> put: description: Update a pod body: <del> json/application: schema: !include doc/pod-schema.json example: !include examples/pod.json </del> <ins> schema: !include doc/pod-schema.json example: !include examples/pod.json </ins> delete: description: Delete a specific pod responses: 200: body: <del> application/json: example: | { "success": true } </del> <ins> example: | { "success": true } </ins> /replicationControllers: get:

Starting a large data set (32gb allocated, 4gb resident) in the api server causes about 7.5gb allocations of . The tree for that shows: This appears to be due to openapi aggregation. If there is an easy fix in 1.7 it might be worth it. Otherwise this should be addressed for 1.8.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
This is all API server startup - for small clusters it's about 20-60% of startup time
Most of those are that is the library we used to convert our spec to proto. The key is this line: Do you have any idea how we can get a better space time/complexity here?
I have a PR for that now.
Fixed in
Even without aggregation, starting and serving OpenAPI docs was a huge chunk of startup time. The linked change makes it disappear.
This should not be related to aggregation at all (as far as I can tell from that tree). Thanks for the fix
Thanks, it's in gnostic master now.


Question: deps = [ ":go_default_library", "//vendor/github.com/onsi/ginkgo:go_default_library", <ins> "//vendor/github.com/onsi/ginkgo/config:go_default_library", "//vendor/github.com/onsi/ginkgo/types:go_default_library", </ins> "//vendor/github.com/onsi/gomega:go_default_library", ], )

Is this a BUG REPORT or FEATURE REQUEST?: /kind flake What happened: TestApps always failed Anything else we need to know?: https://k8s- https://k8s- /sig testing
my pr appear same problem. i will investigate the cause later.
/sig cli
/assign
GitHub didn't allow me to assign the following users: zjj2wry. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fails when run by . Stress testing runs 1000 times locally without any errors. https://k8s-
Hit this again here https://k8s-
this is odd... the test output actually shows TestApps succeeding in that run:
Happened again: https://k8s- The messages were interleaved . was appended at the end. I think that threw off.
By the looks of test grid, it looks like it's been failing consistently in both presubmit and postsubmit builds: https://k8s- https://k8s- The logs are happy, the junit output is not. Maybe it has something to do with the fact that ginkgo is used to run the test cases? It's used for TestOpenapi tests as well. There was an issue on those tests failing for what looks like the same reason Looks like they were fixed by /kind bug /priority failing-test
cc
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 6 days, the issue will be moved out of the v1.8 milestone. priority: Must specify exactly one of [, , ]. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: return nil } <del> glog.V(4).Infof("detach %v from node %q", diskURI, nodeName) </del> <ins> glog.V(2).Infof("detach %v from node %q", diskURI, nodeName) </ins> diskController, err := getDiskController(d.plugin.host) if err != nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Current logging is not enough for azure disk diagnostics, change some log level from 4 to 2 What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign


Question: exec := exec.New() outputBytes, err := exec.Command(nsenterPath, args...).CombinedOutput() if len(outputBytes) != 0 { <del> glog.V(5).Infof("Output from mount command: %v", string(outputBytes)) </del> <ins> glog.V(5).Infof("Output of mounting %s to %s: %v", source, target, string(outputBytes)) </ins> } return err

Currently, there many error messages in that do not contain any context. This means that these messages are basically meaningless in real systems where many pods may be on a node. Examples: Every error message in this file should identify the path that the message is in relation to.
Can you handle this, since you most recently touched in ?


Question: return nil } <del> func testReachabilityOverExternalIP(externalIP string, sp v1.ServicePort, execPod *v1.Pod) error { return testEndpointReachability(externalIP, sp.Port, sp.Protocol, execPod) } </del> func testReachabilityOverNodePorts(nodes *v1.NodeList, sp v1.ServicePort, pod *v1.Pod, clusterIP string) error { internalAddrs := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) externalAddrs := e2enode.CollectAddresses(nodes, v1.NodeExternalIP)

Which jobs are failing: Which test(s) are failing: - Since when has it been failing: Testgrid link: Reason for failure: Nov 4 11:59:41.385: Unexpected error: <*errors.errorString | 0xc00298c3d0: { s: "service is not reachable within 2m0s timeout on endpoint 203.0.113.250:80 over TCP protocol", } service is not reachable within 2m0s timeout on endpoint 203.0.113.250:80 over TCP protocol occurred Anything else we need to know: /priority important-soon /sig network /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Apparently this is related to /cc
if you're busy and want me to take a look tomorrow you can assign that to me and I can deep dive into what's missing/happening :) Happy to help and to understand better the flakiness life
so, the test adds a "fake" external IP that should be nated to the service endpoint It went through the CI, you can check the test here passing more details in the issue I opened explaining the test
ok, this is interesting the test is failing when using container runtime DOCKER https://k8s- the same periodic job that uses container runtime CONTAINERD (same as the PR) is working https://k8s- We need to understand what is different regarding the network in GCE jbos, when using docker as runtime, kubenet??? cc:
one diff between the working jobs with containerd and the failing ones with docker is that the failing ones use this part of the code for reference, kube-proxy adds the NAT rules here for external IPs here
IIRC you are using kubenet for azure jobs, are you using docker as runtime, can you please check if this tests fails there too?
The problem happens on environment that are using the bridge plugin, seems I got wrong the feature will revert until we have the right test.

We use kubenet only for IPv6 and dualstack jobs. Rest of the tests use Azure CNI. Default is docker runtime for all jobs.


Question: if err != nil { // We may encounter errors here because of a race between the pod readiness and apiserver <del> // proxy. So, we log the error and retry if this occurs. return nil, fmt.Errorf("Controller %s: failed to Get from replica pod %s:n%snPod status:n%s", </del> <ins> // proxy or because of temporary failures. The error gets wrapped for framework.HandleRetry. // Gomega+Ginkgo will handle logging. return nil, fmt.Errorf("controller %s: failed to Get from replica pod %s:n%wnPod status:n%s", </ins> controllerName, pod.Name, <del> format.Object(err, 1), format.Object(pod.Status, 1)) </del> <ins> err, format.Object(pod.Status, 1)) </ins> } responses = append(responses, response{podName: pod.Name, response: string(body)}) }

Seen on the CI here in this specific job The apiserver returns 503 and the [function fails inmediatly] () instead of retrying I expected my fix to retry on those errors, but is clearly not the case I find very hard to understand the logic to retry in this function, can you PTAL and identify the reason why that function is not retrying on these errors?
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
As best I can tell if does in fact return true then that should have worked.
The problem is that this code here isn't actually returning a status error: The status error is embedded inside the new error, but not wrapped. Therefore doesn't find it and returns false -no retries. Fix in


Question: }, VirtualServer{}, true, <del> fmt.Sprintf("IPVS Service Flags should be >= %d, got 0x0", FlagHashed), </del> <ins> fmt.Sprintf("IPVS Service Flags should include %x, got 0x0", FlagHashed), </ins> }, { libipvs.Service{

The current support of the load distribution method and in kube-proxy ipvs mode only performs hashing of source IP address to select endpoints. As a result, all connections from a single client (i.e. from a particular source IP) are always forwarded to only one endpoint matching with the hash value of this source IP. If the client opens a vast majority of connections to the k8s cluster, it will result in traffic imbalance as only one endpoint located on one node handles the connections while the remaining endpoints/nodes are idle. Here is an example of the problem when using method for kube-proxy Deploy a service with 4 endpoints located on 4 nodes.Open 200 connections from a client to node IP of any node (e.g. telnet 192.168.1.1 ). All connections receive replies from only one single endpoint.The distribution printout also shows that all connections are forwarded to the endpoint (e.g. 1.0.4.2:5001 in this example)Similar issue is seen with method Connections from a client should be evenly distributed to all endpoints like to avoid unbalanced load distribution between cluster nodes and endpoints. This can be easily done by enabling the source port hash flag that IPVS supports for the methods which performs hashing on both source IP and source port when selecting the target endpoint.
/sig network /area kube-proxy /area ipvs
This issue was raised based on the discussion of the issue
triage/accepted /assign
/triage/accepted
/triage accepted (damn it!)


Question: VerifyDatesInOrder(out, "n" /* rowDelimiter */, "t" /* columnDelimiter */, t) } <ins> func TestDescribeContainers(t *testing.T) { testCases := []struct { input            api.ContainerStatus expectedElements []string }{ // Running state. { input: api.ContainerStatus{ Name: "test", State: api.ContainerState{ Running: &api.ContainerStateRunning{ StartedAt: util.NewTime(time.Now()), }, }, Ready:        true, RestartCount: 7, Image:        "image", }, expectedElements: []string{"test", "State", "Running", "Ready", "True", "Restart Count", "7", "Image", "image", "Started"}, }, // Waiting state. { input: api.ContainerStatus{ Name: "test", State: api.ContainerState{ Waiting: &api.ContainerStateWaiting{ Reason: "potato", }, }, Ready:        true, RestartCount: 7, Image:        "image", }, expectedElements: []string{"test", "State", "Waiting", "Ready", "True", "Restart Count", "7", "Image", "image", "Reason", "potato"}, }, // Terminated state. { input: api.ContainerStatus{ Name: "test", State: api.ContainerState{ Termination: &api.ContainerStateTerminated{ StartedAt:  util.NewTime(time.Now()), FinishedAt: util.NewTime(time.Now()), Reason:     "potato", ExitCode:   2, }, }, Ready:        true, RestartCount: 7, Image:        "image", }, expectedElements: []string{"test", "State", "Terminated", "Ready", "True", "Restart Count", "7", "Image", "image", "Reason", "potato", "Started", "Finished", "Exit Code", "2"}, }, // No state defaults to waiting. { input: api.ContainerStatus{ Name:         "test", Ready:        true, RestartCount: 7, Image:        "image", }, expectedElements: []string{"test", "State", "Waiting", "Ready", "True", "Restart Count", "7", "Image", "image"}, }, } for i, testCase := range testCases { out := new(bytes.Buffer) describeContainers([]api.ContainerStatus{testCase.input}, out) output := out.String() for _, expected := range testCase.expectedElements { if !strings.Contains(output, expected) { t.Errorf("Test case %d: expected to find %q in output: %q", i, expected, output) } } } } </ins> func TestDescribers(t *testing.T) { first := &api.Event{} second := &api.Pod{}

Kubectl describe pods id only has top level pod information, without any information related to containers. Taking kube-dns pod as an example, which has 3 containers specified: If a pod with RestartAlways policy, but all contianers are in crashloop, the user cannot interpret the true state based on above information. Proposed to include ContainerStatus here.
Why isn't this redundant with ?
is for kubectl get pods id.


Question: } // We have unmamanged pods, instead of erroring and hotlooping in disruption controller, log and continue. if len(unmanagedPods) > 0 { <del> klog.V(4).Infof("found unmanaged pods associated with this PDB: %v", strings.Join(unmanagedPods, ",'")) </del> <ins> klog.V(4).Infof("found unmanaged pods associated with this PDB: %v", unmanagedPods) dc.recorder.Eventf(pdb, v1.EventTypeWarning, "UnmanagedPods", "Pods selected by this PodDisruptionBudget (selector: %v) were found "+ "to be unmanaged or managed by an unsupported controller (missing a /scale subresource). As a result, the status of the PDB "+ "cannot be calculated correctly, which may result in undefined behavior. To account for these pods please set ".spec.minAvailable" "+ "field of the PDB to an integer value.", pdb.Spec.Selector) </ins> } currentTime := dc.clock.Now()

When using PDBs with or (percentage), it is not supported to reference pods via a PDB selector, that do not have a supported controller owner reference: Deployment, ReplicationController, ReplicaSet, and StatefulSet controllers or a resource implementing subresource This is also documented Nevertheless, there are still people who create PDBs a pods without controllers that ignore this. For example PDBs that set , want to block eviction, and node drain (it works for them but the PDB status is invalid). Unfortunately there is no feedback for the user to receive when this happens. The KCM logs is the only place that informs the user when unmanaged pods are found. Which I think is insufficient. There should be some way to inform users about this. Since it cannot be know on admission we probably have to do it when reconciling the PDB and unmanaged pods are found. Options: an event when this is detected - probably needs to be throttled in some way a condition in a PDB status that would be set and admins could monitor these to detect misbehaving PDBs create a PDB create unmanaged pod observe incorrect status because this configuration is unsupported No response <detailsNA </details<detailsNA </details<detailsNA </details<details</details<details</details<details</details
/sig apps
/triage accepted This was discussed during today's sig-apps, we've agreed to add a warning event right next to the log statement from the original description
I'll take this on.
/assign
/reopen I did some more testing for the unsupported controllers and found out that these are covered by the event, so I posted a followup to reflect that The new event should cover only the naked pods
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return PolicyRule{}, fmt.Errorf("non-resource rule may not have apiGroups, resources, or resourceNames: %#v", r.PolicyRule) } case len(r.PolicyRule.Resources) > 0: <del> if len(r.PolicyRule.NonResourceURLs) != 0 { return PolicyRule{}, fmt.Errorf("resource rule may not have nonResourceURLs: %#v", r.PolicyRule) } </del> <ins> // resource rule may not have nonResourceURLs </ins> if len(r.PolicyRule.APIGroups) == 0 { // this a common bug return PolicyRule{}, fmt.Errorf("resource rule must have apiGroups: %#v", r.PolicyRule)

validated in case len(r.PolicyRule.NonResourceURLs) 0 is useless. Just for merge. ref to
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig auth
I think this issue can be closed now. has been merged and solves this.


Question: klog.Infof("ExpandVolume.UpdatePV succeeded for volume %s", pvcWithResizeRequest.QualifiedName()) } <ins> fsVolume, _ := util.CheckVolumeModeFilesystem(volumeSpec) </ins> // No Cloudprovider resize needed, lets mark resizing as done // Rest of the volume expand controller code will assume PVC as *not* resized until pvc.Status.Size // reflects user requested size. <del> if !volumePlugin.RequiresFSResize() { </del> <ins> if !volumePlugin.RequiresFSResize() || !fsVolume { </ins> klog.V(4).Infof("Controller resizing done for PVC %s", pvcWithResizeRequest.QualifiedName()) err := resizeMap.MarkAsResized(pvcWithResizeRequest, newSize)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created a block EBS volume using dynamic provisioning. Then I resized the volume from 10GB to 20 GB. Both PV and block volume reflects the change, but the PVC's capacity is still 10GB. This could be caused by in-tree resize put the PVC condition under state, but no one is acting on the condition since it is block volume: /cc /sig storage What you expected to happen: PVC's capacity should be updated to 20GB How to reproduce it (as minimally and precisely as possible): Easy to reproduce using following specs: StorageClass: Claim: Pod: Anything else we need to know?: This will affect whether CSI migration should handle migration scenario for block volume resizing. Environment: Kubernetes version (use ):1.13 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: kops Network plugin and version (if this is a network-related bug): Others:
/sig aws
/assign


Question: "net" "net/http" "os" <ins> "regexp" </ins> "strconv" "strings" "time"

Related with issue , the API server should have a cap on the number of outstanding operations/requests it's handling, and return an error otherwise. A simple semaphore is an idea for this.
+1 !
The good news/bad news on this is that there is a rate limiter in the API server (), but the defaults for it seem too high: 10 qps average, 200 burst (). We could start by lowering the burst level to something smaller, like 30. It may be interesting to also differentiate between GET and PUT/POST requests.
I think that we need limits on the number of outstanding requests, not the number of QPS.
-An max outstanding would be useful. But it may force us to have separate buckets for different types of requests, i.e. a small number of long running requests causing API server to reject short requests. -Heard that you might be looking into adding support for flow control (or something like it?). We may need it soon to help avoid bugs like runaway node status updates (see issue ).
I had not - the things we were looking at were the bastion changes to apiserver which will break logs and proxying into their own rest resources so they can be granularly controlled. Being able to timebox and classify requests differently would be good. ----- Original Message -----
- Ah, np, I probably misheard.
What are our plans for handling this client side? Do we want clients to handle "you're being throttled" error themselves, or do we plan to handle it in the library? I'm asking because currently kubelet does not have a 'short' retry loop, hence it tries to report NodeStatus every X (2) seconds, whether it succeed or not. Neither we have a sidechannel for heartbeat messages. This would mean, that if API server will be under heavy load, and with not-negligible probability NodeStatus reporting will be throttled, we may incorrectly mark a Node as unreachable. Currently it's not a huge problem, because X << Y, where Y is the time we wait before we decide a Node is unreachable. But because of the effort somewhere close to it'll probably get worse.
The client library should have options for retry on client.Config which include the ability to disable the behavior. Automatic retry is probably an "opt-out" behavior.

----- Original Message -----
says there are people working on rate limiting client connections to API server.
I have in the past adjusted the rate limiter to support the creation of large clusters. I also the logic. If you adjust the rate limiter settings, please make sure it is still possible to create large clusters.
I'm closing this as fixed, given the original subject. If we want to expand client-side support for retry, please open a dedicated issue. (Also note that QPS based throttling was to the client recently too)


Question: return suggestedDelay, utilerrors.NewAggregate(errs) } <del> // makeFlowSchemaConditionPatch takes in a condition and returns the patch status as a json. func makeFlowSchemaConditionPatch(condition flowcontrol.FlowSchemaCondition) ([]byte, error) { o := struct { Status flowcontrol.FlowSchemaStatus `json:"status"` }{ Status: flowcontrol.FlowSchemaStatus{ Conditions: []flowcontrol.FlowSchemaCondition{ condition, }, }, } return json.Marshal(o) </del> <ins> func apply(client flowcontrolclient.FlowSchemaInterface, fsu fsStatusUpdate, asFieldManager string) error { applyOptions := metav1.ApplyOptions{FieldManager: asFieldManager, Force: true} // the condition field in fsStatusUpdate holds the new condition we want to update. // TODO: this will break when we have multiple conditions for a flowschema _, err := client.ApplyStatus(context.TODO(), toFlowSchemaApplyConfiguration(fsu), applyOptions) return err } func toFlowSchemaApplyConfiguration(fsUpdate fsStatusUpdate) *flowcontrolapplyconfiguration.FlowSchemaApplyConfiguration { condition := flowcontrolapplyconfiguration.FlowSchemaCondition(). WithType(fsUpdate.condition.Type). WithStatus(fsUpdate.condition.Status). WithReason(fsUpdate.condition.Reason). WithLastTransitionTime(fsUpdate.condition.LastTransitionTime). WithMessage(fsUpdate.condition.Message) return flowcontrolapplyconfiguration.FlowSchema(fsUpdate.flowSchema.Name). WithStatus(flowcontrolapplyconfiguration.FlowSchemaStatus(). WithConditions(condition), ) </ins> } // shouldDelayUpdate checks to see if a flowschema has been updated too often and returns true if a delay is needed.

Integration test timed out due to the following error The error appeared 748 times in the log. Looks like lots of churning happens during the apf configuration bootstrapping phase. Is there a race condition between when the type is being registered and the API calls? Stack trace: Integration job link: PR: I searched for this error in CI and this PR, looks like we only see this error in this PR so far. So I am hoping it has not introduced any flake. (maybe we need to let more time to pass to start seeing flakes) We should not see this error I saw it only in the PR mentioned.
/sig api-machinery cc
I see
The fact that it happens to a non-mandatory object implies that the object was received from the server, so the type must be in the server. The APF config-consuming controller does this:
Look for the output from this line:
Another PR with this failure:
the panic means the request scope FieldManager is nil that is initialized here: OpenAPIModels is set here: called from here: and openAPIModels is constructed here: it looks like it is possible for openapiconfig to be unset, which causes the openapi models to be nil, which causes apply to panic. I don't know if it is only the integration tests that are subject to that configuration. cc /priority important-soon (since this is now being exercised in bring-up of integration test servers)
if APF is creating/bootstrapping the objects without using apply, and then updating status using apply, that would explain how the object exists but apply fails
I suspect that APF is not successfully updating status at all in the integration tests with this server configuration
do you want me to revert , while we work on a solution?
I don't know how bad the test impact is. Work with / to determine if we should revert or fix forward
: I do not follow why you talked about server-side stuff then suggested the problem is in the client (a controller in the kube-apiserver) not supplying a fieldManager. The client does supply the fieldManager, for both creates and updates. See the following lines:
Different field manager. is set server-side in the endpoint handler based on the server configuration.
So if the problem is server-side, why did you suggest the problem is in the client?

Regarding the client, I looked in and found this line: actually several of the same form, all with the same fieldManager.
I don't think I did suggest that :) all of the setup and calls in are server-side
: please help me understand . That is about the client side. Are you saying that a client that creates an object using and later updates the object using --- with a non-empty fieldManager in both --- will run into this server-side scenario? If so, is that because this client behavior is wrong or because there is a bug server side?
That was in response to , trying to figure out how the object was created but could not be updated. Yes, when the server is not configured to enable openapi and therefore does not have apply enabled. Because the server is misconfigured (which, from what I can tell, is limited to some of the server instances started for integration tests)
Looking at and it is not obvious to me whether or how openapi could be disabled. Would that be by turning off the feature flag? FYI, the inclusion of "V3" in the flag makes me unsure what the server does about other versions. : I think you are saying that it is possible to disable openapi and that this in turn disables server-side apply. That seems kind of surprising to me. Does this mean that it is simply incorrect to run a client that does server-side apply with a server that has openapi disabled? Is this intended? Is it documented anywhere?
It's not intended as an externalized configuration option. integration tests have the ability to fiddle with many of the config struct options in the server that a kube-apiserver binary invocation could not. They intentionally use this to do things like inject specialized authorizers / authenticators for testing purposes, but also can unintentionally result in a test server that doesn't match a real API server invocation. I think this is an integration test server configuration problem, which should be fixed.
is correct that we have a misconfiguration in the integration tests and SSA is only enabled on a subset of integration tests. I'd like to understand the test failure conditions a bit better though. The can be consistently reproduced on a non SSA targeted test. (Eg: selected from one of the test failures). However, I'm seeing the panic message while the test is still green and passes locally.

it's a bit bizarre what contributes to the flakiness of the test because the FieldManager panic can be reproduced 100% of the time.
The panic occurs when APF is updating status (which it just switched to do using SSA). The APF status update failure does not block the API requests the test cares about, so the test passes. I could see extra CPU usage contributing to flakiness.
Thanks for the explanation. Created to fix the integration test configurations.
/triage accepted
The integration test job started timing out dramatically last night (50% of runs) once merged. Since that was the only change made to integration tests recently, it seems related. I think we should revert (and therefore ) until the test impact is understood
revert of -
did this get fully reverted? we're still seeing panics in integration tests over this:
bumping priority and milestoning until we know we're not currently triggering panics
looks like it got reintroduced in , revert open in
For integration tests, maybe we can cheat and cache the OpenAPI data, so that it's not recomputed for each test?
:-/ maybe... but I really don't want to have non-real init paths in integration if at all possible
I wish we could go a little further in lazily initializing openapi (especially v2), so let's see what we can do here (after code-freeze probably)
if SSA is actively using the openapi, it being lazy won't help, right?
does the revert mean that we can remove milestone v1.24 from this issue?
Yes, cleared
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Taking a fresh look at this PR and wondering if it has been mostly fixed already? From comments my understanding is that this panic occurred due to two conditions: running as part of the integration test suite is asked to service an SSA request while are nil. The apiservice has logic to panic in this case. From the code path commented is nil when the server is created with is nil. APF worker (running alongside an integration test) issues an SSA request to the test apiserver. Since the apiserver is misconfigured it panics. This panic could happen sporadically because sometimes the tests finish before the APF worker is able to issue the request. As a brief check to see if this might be fixed, I ran the test mentioned was failing from SSA. I logged out the 's created and found that they all had managed fields. I saw this as an indication SSA was now functioning properly. Indeed, to start the test server the function calls . This function has a code path which unconditionally leads to the apiserver's being set. calls before running the returned apiserver: calls before using the returned config to create the returned apiserver: uses which sets : Thus, any test using should no longer be affected by this panic. The question then becomes: do all integration tests start their servers in this way? It's hard to say. I found a PR (, ) which refactors many occurrences of the old stanza: Into something using the new method: ` But it appears is still being used in a few integration tests: Should these occurrences be refactored to use too?
There was a ton of changes I did around that. But the last PR (already removing the old is already out for review): - FYI Thinking about it now, I guess I agree that the problem might have been 10 different way kube-apiserver was set up through different tests.

I just don't know why it was causing flakes, not failing consistently...
There were definitely test setups that were not enabling openapi in the recent past. Two possibilities for flakes: Enabling openapi in all integration test setups was expensive and caused spurious timeouts The excessive numbers of background panics handled and logged in integration tests when the API server tried to handle SSA requests bogged down the CI machine (I think something like 40,000 panics with full stack traces each were getting logged)
i don't think that's the case. With the recent changes that I was doing as part of I think OpenAPI is already enabled for all tests having kube-apiserver (almost all) and that seems to work. But you remainded me that back then we also had an issue that we were running many more tests simulanuously, which could have been contributing to this problem significantly. The second reason is definitely possible contributor too. I guess my concern was that if some requests were panicing due to lack of OpenAPI, this means that these aren't really needed. But that actually might make sense - the fact that we weren't able to upgrade status of PriorityLevels/FlowSchemas shouldn't prevent non P&F-tests from succeeding, and that worked in P&F tests...
I think the panics were happening in async controllers, and no functionality blocked by the controllers intersected with the surface being tested, it just (maybe) caused log/perf issues
I believe has resolved the panic issue, I was able to verify with which changes the priority & fairness controller to use server side apply. Before merged, I analyzed the integration log: Integration test log file from an unrelated pr does not contain any trace of the panic as expected: On the other hand, the log file from integration job for (before merged) had plenty of these panics even though the integration job turned green: I wanted to check which tests produced these panics, so I did a breakdown by package Then I narrowed it down to the individual test(s): Before merged, the above tests used , and either is or is which results in to be nil. But a quick code search revealed that many more integration tests used , so why were the other tests not affected by the panic?

I inspected the log of a test that was not affected by the panic, and grepped for , and found the following: In the above case, the apf controller had not advanced far enough, the controller was active for about , may be these tests finish quickly and signals the server to shutdown before the apf controller can send an APPLY command. On the other hand, log from a test that was affected by the panic shows that the apf controller was active for more than , and has lot more activities: I am assuming that integration tests have at least log level 4. Also, the size of the log file differed significantly, without panics around , and with panics over and changed all these tests to use and now integration log for shows no panic.
Great! (This also proves why eliminating gazilion different things of initializing stuff in favor of one makes sense). - FYI ^^ - let's merge the PR then - I just one question there, so let's wait for Jordan to respond it.
Can we just force the openapi/fieldmanager to be set at start-time now that SSA is GA, it should basically always be present, rather than fail when SSA is unavailable (that's probably wrong by the way). If that triggers any test flakiness, then we would solve these. We can wait for to be solved since it will almost certainly have a huge impact on this.
Has this been done Jeff?


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // Package buffered provides an implementation for the audit.Backend interface // that batches incoming audit events and sends batches to the delegate audit.Backend. package buffered // import "k8s.io/apiserver/plugin/pkg/audit/buffered" </ins>

We should introduce a buffer (similar to what we have in the webhook output backend) to the log file output backend. Audit events should be queued on a relatively large buffer chanel, and an output go routine should grab events off the queue and write them to disk. If an audit event is output when the channel is full, it should be dropped (this should be counted by a prometheus metric, and an e2e test should ensure that the count is below some threshold). This behavior should be optional (configurable) /cc is tracking the regression
Naturally os will cache , these cached data will be written to disk either by fsync or pdflush periodically. What do you think is the most time-costing procedure in audit?
The problem is not with the time consumed. The problem is that currently write is performed synchronously with processing the request, which slows the request down, no matter how well caching works
The problem is that currently write is performed synchronously with processing the request Sure synchronously write does cost time serializing. Introduce buffer mechanism can really reduce api request latency time.
It doesn't matter that much if it happens asynchronously Indeed
Yes, this is a good point to improve performance.
Now I totally comprehend this. I am pleased to implement this buffer.
I'll be happy to review that
audit log is important sensitive information which we should try to make sure it reaches disk at all cost. Does this effort compromises that in favor of optimizing for latency ?
I'll make sure it's configurable and if the security is critical, you can setup synchronous write in your deployment. Having said that, it's critical to be able to accommodate the case where having 100% calls written does not compensate for increased latencies. Does it make sense?
thanks my preference would be to keep the default as no buffering when making it configurable. I didnt understand how does one setup "synchronous write in your deployment" and how that will affect audit logs reaching disk. My worry is mainly around a crash where the buffered audit logs dont reach the disk and are lost
if use buffer, audit events lost is inevitable when crash. As like webhook backend, still exist this risk. But to improve apiserver qps, something should comprise. By the way, we still reserve the synchronous write mode, which is configurable.

To make it simple, when offered trade-off, people are more likely to give up audit logs reliability than performance. Makes sense?
i am not against this change. i think what would be best is to make the default as no buffering and enable buffering through a parameter. Also for Enterprises, the trade off is not correct, they would prefer to NOT give up audit logs
I understand what you're suggesting and I don't agree :) Switching to buffering will for sure increase the latency, but will only increase the probability of losing logs, which will never be zero. I'm pretty sure we need to have buffered mode by default. However, since this argument is about making a trade-off and since neither of us represents quorum, let's ask more people for their opinion What do you think?
The default is slightly less important here since audit logging as a whole is not enabled by default. If we do turn it on by default, it becomes more important. That said, my hunch is that we should default to the mode with lower performance impact, but I don't have strong feelings either way.
As long as there are options to turn it on of off (the synchronous write) I'm fine with either default. Still, I'd like to see due diligence towards ensuring low lost rate when using buffered write, and sufficient documentation around caveats of using one or the other. Audit, from my personal experience, is crucial for majority of enterprises and they won't tolerate not even single lost event.
That is not possible, there will always be a situation when an audit event can be lost: sudden power loss, disk/network failure, random bit flip, etc. The questions is whether 10^(-6) is low enough loss probability, or we need to get 10^(-14) by sacrificing performance (arbitrary numbers, just to illustrate my point). Note, that reduced performance can increase the number of timeouts and retries, increasing load and leading to failures and even more lost events.
I think this is an important bug that we're not testing audit logging at scale
Marking it as urgent, since it's blocking
If split into 3 prs, I must do it one by one . Now first one is up .
SGTM Status of this issue: ETA: 1/12/2017 Risks: Complicated fix required
has been reviewed for almost 2 months and has not done.

If I raise 3 prs to replace it, I am still worried whether all of them can be merged. They act as a whole.
Now I'm going to devote my time to it, we'll do it
OK.
In progress, reviewed first PR, waiting for applying changes ETA: Dec 01
Per Exception discussion, this portion of the feature is being removed from the milestone. As such, changing the priority. /priority important-soon /remove-priority critical-urgent
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: "x-kubernetes-patch-strategy": "merge" }, "hostAliases": { <del> "description": "HostAliases is an optional list of hosts and IPs that will be injected into the pod's hosts file if specified. This is only valid for non-hostNetwork pods.", </del> <ins> "description": "HostAliases is an optional list of hosts and IPs that will be injected into the pod's hosts file if specified.", </ins> "items": { "allOf": [ {

there is a note above hostAliases that this feature does not work with hostNework=true, yet it seems it does: when testing with a hostNetwork pod + hostAliases, there are no errors in kubelet logs at --v=10 related to this, in fact i don't see any hostAlias entries at all in there. i could not find anything blocking this in the kubelet, in fact there is normal handling of hostNetwork: i could not find any validation performed on the hostAliases field of PodSpec if hostNetwork is true on the API level. i could not find any unit tests or e2e tests related to this. the godoc should contain valid information if the feature is not supposed to work, it must be patched to not have the entries in the hosts file (perhaps?) we should have tests that ensure this type of requirement is guaranteed these steps confirm that a static pod with hostNework (deployed by kubeadm) does make use of hostAliases. create a kubeadm single node cluster directly with or one with modify the to include:wait some time for the new static pod spec to resync find the PID of the kube-apiserverget the hosts file n/a latest master 1.30.0-alpha.0 not applicable <details</detailskubeadm containerd v1.6.14 kindnet add test: update godoc:
/sig node network cluster-lifecycle /kind documentation
as reported by the hostAliases sometimes are not mounted for hostNetwork static pods, which is concerning: related slack thread:
At minimum the comment is wrong I need to be updated and we need e2e tests :/
found this, support was a while back: i will PR the API docs.
We identified an edge case here with Kubernetes v1.29. If a cloud provider did not yet initialise the node's , the code for creating the file including the hostAliases does not get run. To be concrete, in: is empty in this case which leads to to be . I was able to trace it back to this change which leads to this behaviour for static pods: For confirmation: I reverted the above commit, rebuilt kubelet and it got back to the old behaviour.

there is no perfect solution, but reverting is wrong because opens another bug, we need to study this more carefully but in the meantime installers that know the nodes addresses beforehand should set the --node-ip flag or not use cloud-provider external
yeah, as i mentioned downstream, revering is out of the question. it fixes some long standing problems. we might have to document somewhere this limitation in a cloud provider env.
i think it might be possible for us to special case this so that the hostAlias management does not wait for the node IP in a CP env, under certain conditions. just a speculation.
PRs add test: update godoc:
/triage accepted looks like it is being worked on already


Question: } return path, nil } <ins> // deduplicate removes any duplicated values and returns a new slice, keeping the order unchanged func deduplicate(s []string) []string { encountered := map[string]bool{} ret := make([]string, 0) for i := range s { if encountered[s[i]] { continue } encountered[s[i]] = true ret = append(ret, s[i]) } return ret } </ins>

/kind bug /sig cli What happened: The same path was multiple times to KUBECONFIG env variable. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-05T00:40:04Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"darwin/amd64"} Cloud provider or hardware configuration**: Irrelevant OS (e.g. from /etc/os-release): MacOS Sierra 10.12.6 Kernel (e.g. ): Darwin Kernel Version 16.7.0 Install tools: brew install kubernetes-cli Others:
I would help fix it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: if apiServiceInfo, ok := s.apiServiceInfo[apiservice.Name]; ok { apiServiceInfo.apiService = *apiservice apiServiceInfo.handler = handler <ins> return </ins> } s.apiServiceInfo[apiservice.Name] = &openAPIV3APIServiceInfo{ apiService: *apiservice,

When running an extension apiserver, the resouces and groups served by this apiserver are missing from openapi/v3 discovery from time to time (approx 1 in 100 requests). the requests should pass 100% of the time repeat 100 times and end up with we can see it is missing in the openapi/v3 endpoint No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/triage accepted /cc


Question: err = proxier.iptables.RestoreAll(proxier.iptablesData.Bytes(), utiliptables.NoFlushTables, utiliptables.RestoreCounters) if err != nil { klog.Errorf("Failed to execute iptables-restore: %vnRules:n%s", err, proxier.iptablesData.Bytes()) <ins> metrics.IptablesRestoreFailuresTotal.Inc() </ins> // Revert new local ports. utilproxy.RevertPorts(replacementPortsMap, proxier.portsMap) return

What would you like to be : kube-proxy should expose better metrics around tracking success and failures. Administrators should be able to alert on failure counts. Why is this needed: There are several motivating cases: iptables lock contention causes to fail at an increasing rate as services + endpoints grows or are failing to execute Ultimately, there is essentially no feedback from kube-proxy if it fails to program the service network. This makes it annoyingly difficult to debug when things go wrong. As many reasonable hints we can provide should be . /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
is anyone working one this..
what cases do you think are poorly covered? /assign
thanks for triaging this :-) The specific cases are when fails - that function logs an error but doesn't otherwise surface any information. We should have a top-level metric for failing, along with any other lower-level metrics that might make sense for administrators, e.g. fails, fails.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/help-wanted
/assign


Question: // JobStatus represents the current state of a Job. message JobStatus { // The latest available observations of an object's current state. <ins> // When a job fails, one of the conditions will have type == "Failed". </ins> // More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/ // +optional // +patchMergeKey=type

The API docs for jobs are unclear as to what fields a caller would use to determine job success or failure in 1.11 (and we haven't updated them in a while) A casual reader would infer that completionTime is set when the job fails (but it isn't). Linking out to the docs is ok, but I shouldn't have to click through the links to understand how to use the Job object. The Job description and the job status fields should clearly indicate what fields are set in completion states on the job, because that's 100% of the point of docs. Taking a summary of the jobs-run-to-completion link and putting it in , , and job would improve the actual utility of our API doc. We should also do a pass on other objects and make the actual in server docs more useful.
as a consumer of the API, I had to read way too much to know how to use jobs correctly and how to write an API client that properly handled the conditions. Also, we should consider either setting completionTime on failures, or adding failedTime (the former may not be forwards compatible), because reading the conditions is too complex for a fundamental use case like this. This matches our decision w.r.t. conditions vs. fields - conditions are useful for extension and extra data, but status should have fields that allow an API consumer to easily understand state.
Also... our docs don't actually explain how to determine the job is complete via the API, so there's arguably no place this is captured.
I have the same problem right now, of correctly identifying a finished job, when it does not complete successfully and after reading the documentation and the reference and all the examples I could find, it's still not clear how I can check the conditions and make sure the job is finished when I do so: -what does each mean? can I rely on , is that set only when the job finishes, whether via or ? what other reasons are possible? It would definitely help to have specified on failure as well, and the differentiation to be made via the presence of .
/assign soltysh
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I have to say this issue is still very relevant. As of now, it is very hard to design something around Job Statuses. Anyway to move this forward?
Completely agree with previous comments! Even I found the way how to detect completed/failed jobs, still not sure am I covered all use cases. This part definitely needs clarifications/improvements/better documentation.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
With regards to: If you actually follow this link and then search on that page for "conditions" you will get exactly zero hits. This is extremely unhelpful.
This documentation really, really needs work. (So does the API, for that matter.)
So what is the correct way to determine if a job is finished and whether it completed successfully?
When a Job completes (finished successfully), its will be set, and Job conditions will have one with . When a Job fails, Job conditions will have one with .
Hi, if nobody has been working on it, can I take this one? I have gone through the issue which is mentioned here and I would like to work on this. Can you please help me to get started as I am beginner to the project.
I totally agree with this- currently the only way to get the full picture of a Job is to read its conditions along with status fields which is error prone and a lot of extra work. I think a good start would be to add a new field as suggested by so as to not break compatibility for users already depending on
This sound reasonable. I'm still unclear when can contain more than one element. Also what does a or mean?
Still, open -- anyone actively working on this? I would like to attempt if not. :)
Are you still working on this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
bump
This does not match what I just witnessed in my cluster… digging deeper to see if I need to file a bug but I have a Job with a single Pod that was executed. The Pod failed but the Job was marked as complete. This is the one (and only) pod that executed as my job: And this is the job that owns it (from kubectl describe): This is the job status object: on v1.16.7 but not finding any similar reports. just found this page as my only relevant search result.
I think it not only needs documentation, but also a change: It would be good to have 2 fields (probably needs better names), this way it's easy and clear when a job is finished and if it failed or not and when (someone can look at failed, succeeded and conditions for details): finishedtime: only set when the job finished (either completely failed or completely finished) finishtype: 'failed or completed'


Question: "io/ioutil" "os" "regexp" <ins> "strings" </ins> "time" "github.com/Azure/azure-sdk-for-go/services/containerregistry/mgmt/2019-05-01/containerregistry"

What happened: An aks-engine cluster with a user assigned managed service cluster identity fails to authenticate to azure container registries in private clouds when the proper (acrpull) azure role assignments are in place and when there are no image pull secrets. What you expected to happen: The cluster should be able to authenticate and pull from an ACR hosted in the same azure cloud without needing additional pull secrets when the cluster identity has the proper acrpull azure role assignment, no matter which azure cloud it is operating in. How to reproduce it (as minimally and precisely as possible): Assume you have a private azure cloud where every url ends with "". That would mean the ACR suffix would be . Let's pretend you have an ACR at and you try to deploy a pod in your cluster referencing the image . This will fail to pull when you have no image pull secrets. Anything else we need to know?: The problem stems from the following lines where expected ACR suffix domains are hardcoded: Environment: Kubernetes version (use ): 1.16.9 Cloud provider or hardware configuration: Azure private OS: ubuntu 16.0.4 Network plugin and version: Azure CNI
/sig provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure /sig cloud-provider
It looks like we may be able to obtain the acr suffix from the azure environment:
there is a security issue here: if we support self defined ACR domain, e.g. , then service principal secret could be sent to that remote server. The reason why we use hardcoded domains is that we could always make sure the following 4 domains are owned by Azure:


Question: {ID: "AThirdDeviceId", Health: pluginapi.Healthy}, } <del> p, e := esetup(t, devs, socket, "mock", func(n string, a, u, r []pluginapi.Device) { require.Len(t, a, 1) require.Len(t, u, 1) require.Len(t, r, 1) require.Equal(t, a[0].ID, updated[1].ID) require.Equal(t, u[0].ID, updated[0].ID) require.Equal(t, u[0].Health, updated[0].Health) </del> <ins> callbackCount := 0 callbackChan := make(chan int) callback := func(n string, a, u, r []pluginapi.Device) { // Should be called twice: // one for plugin registration, one for plugin update. if callbackCount > 2 { t.FailNow() } // Check plugin registration if callbackCount == 0 { require.Len(t, a, 2) require.Len(t, u, 0) require.Len(t, r, 0) } // Check plugin update if callbackCount == 1 { require.Len(t, a, 1) require.Len(t, u, 1) require.Len(t, r, 1) require.Equal(t, a[0].ID, updated[1].ID) require.Equal(t, u[0].ID, updated[0].ID) require.Equal(t, u[0].Health, updated[0].Health) require.Equal(t, r[0].ID, devs[1].ID) } callbackCount++ callbackChan <- callbackCount } </ins> <del> require.Equal(t, r[0].ID, devs[1].ID) }) </del> <ins> p, e := esetup(t, devs, socket, "mock", callback) </ins> defer ecleanup(t, p, e) go e.run() <ins> // Wait for the first callback to be issued. <-callbackChan </ins> p.Update(updated) <del> time.Sleep(time.Second) </del> <ins> // Wait for the second callback to be issued. <-callbackChan </ins> e.mutex.Lock() defer e.mutex.Unlock()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: For now, UT in doesn't work because callback function will never be called, is rewritten in this function : Another thing, i'm wondering why we rewrite device health in function which in this file: What you expected to happen: UT assertion should be executed. How to reproduce it (as minimally and precisely as possible): Add some logs in callback function of , it will show nothing. Anything else we need to know?: I will send a PR to fix this. /cc WDYT? /sig node
/assign
GitHub didn't allow me to assign the following users: RenaudWasTaken. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Good catch


Question: --project "${PROJECT}"  --zone "${ZONE}"  --machine-type "${MINION_SIZE}"  <ins> --boot-disk-type "${MINION_DISK_TYPE}"  --boot-disk-size "${MINION_DISK_SIZE}"  </ins> --image-project="${IMAGE_PROJECT}"  --image "${IMAGE}"  --tags "${MINION_TAG}"

The node disk type should be an option in the cluster/gce/config-*.sh files. The nodes get created here:
And also the disk size.
How to do this for aws ? I am stuck at 10GB minion
It . A pull request adding it would be very welcome!
please check this thread
Hello there, can I do something to make this available in the command ? I'd like to us an SSD drive :)


Question: func Register() { // Register the metrics. registerMetrics.Do(func() { <del> for _, metric := range metricsList { legacyregistry.MustRegister(metric) } </del> <ins> RegisterMetrics(metricsList...) </ins> volumeschedulingmetrics.RegisterVolumeSchedulingMetrics() PodScheduleSuccesses = scheduleAttempts.With(metrics.Labels{"result": "scheduled"}) PodScheduleFailures = scheduleAttempts.With(metrics.Labels{"result": "unschedulable"})

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Support adding extra metrics for framework plugins Why is this needed: /sig scheduling
/assign