Question: func (fakeRL) Stop()             {} func (f fakeRL) CanAccept() bool { return bool(f) } <ins> func expectHTTP(url string, code int, t *testing.T) { r, err := http.Get(url) if err != nil { t.Errorf("unexpected error: %v", err) } if r.StatusCode != code { t.Errorf("unexpected response: %v", r.StatusCode) } } func TestMaxInFlight(t *testing.T) { const Iterations = 3 block := sync.WaitGroup{} block.Add(1) sem := make(chan bool, Iterations) re := regexp.MustCompile("[.*/watch][^/proxy.*]") server := httptest.NewServer(MaxInFlightLimit(sem, re, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if strings.Contains(r.URL.Path, "dontwait") { return } block.Wait() }))) defer server.Close() // These should hang, but not affect accounting. for i := 0; i < Iterations; i++ { // These should hang waiting on block... go func() { expectHTTP(server.URL+"/foo/bar/watch", http.StatusOK, t) }() } for i := 0; i < Iterations; i++ { // These should hang waiting on block... go func() { expectHTTP(server.URL+"/proxy/foo/bar", http.StatusOK, t) }() } expectHTTP(server.URL+"/dontwait", http.StatusOK, t) for i := 0; i < Iterations; i++ { // These should hang waiting on block... go func() { expectHTTP(server.URL, http.StatusOK, t) }() } // There's really no more elegant way to do this.  I could use a WaitGroup, but even then // it'd still be racy. time.Sleep(1 * time.Second) expectHTTP(server.URL+"/dontwait/watch", http.StatusOK, t) // Do this multiple times to show that it rate limit rejected requests don't block. for i := 0; i < 2; i++ { expectHTTP(server.URL, errors.StatusTooManyRequests, t) } block.Done() // Show that we recover from being blocked up. expectHTTP(server.URL, http.StatusOK, t) } </ins> func TestRateLimit(t *testing.T) { for _, allow := range []bool{true, false} { rl := fakeRL(allow)

Related with issue , the API server should have a cap on the number of outstanding operations/requests it's handling, and return an error otherwise. A simple semaphore is an idea for this.
+1 !
The good news/bad news on this is that there is a rate limiter in the API server (), but the defaults for it seem too high: 10 qps average, 200 burst (). We could start by lowering the burst level to something smaller, like 30. It may be interesting to also differentiate between GET and PUT/POST requests.
I think that we need limits on the number of outstanding requests, not the number of QPS.
-An max outstanding would be useful. But it may force us to have separate buckets for different types of requests, i.e. a small number of long running requests causing API server to reject short requests. -Heard that you might be looking into adding support for flow control (or something like it?). We may need it soon to help avoid bugs like runaway node status updates (see issue ).
I had not - the things we were looking at were the bastion changes to apiserver which will break logs and proxying into their own rest resources so they can be granularly controlled. Being able to timebox and classify requests differently would be good. ----- Original Message -----
- Ah, np, I probably misheard.
What are our plans for handling this client side? Do we want clients to handle "you're being throttled" error themselves, or do we plan to handle it in the library? I'm asking because currently kubelet does not have a 'short' retry loop, hence it tries to report NodeStatus every X (2) seconds, whether it succeed or not. Neither we have a sidechannel for heartbeat messages. This would mean, that if API server will be under heavy load, and with not-negligible probability NodeStatus reporting will be throttled, we may incorrectly mark a Node as unreachable. Currently it's not a huge problem, because X << Y, where Y is the time we wait before we decide a Node is unreachable. But because of the effort somewhere close to it'll probably get worse.
The client library should have options for retry on client.Config which include the ability to disable the behavior. Automatic retry is probably an "opt-out" behavior.

----- Original Message -----
says there are people working on rate limiting client connections to API server.
I have in the past adjusted the rate limiter to support the creation of large clusters. I also the logic. If you adjust the rate limiter settings, please make sure it is still possible to create large clusters.
I'm closing this as fixed, given the original subject. If we want to expand client-side support for retry, please open a dedicated issue. (Also note that QPS based throttling was to the client recently too)


Question: if g.kubeScheduler == "" { return SchedulerMetrics{}, fmt.Errorf("kube-scheduler pod is not registered. Skipping Scheduler's metrics gathering") } <del> output, err := g.getMetricsFromPod(g.client, metricsProxyPod, metav1.NamespaceSystem, kubeSchedulerPort) </del> <ins> var err error var output string g.waitForSchedulerReadyOnce.Do(func() { var lastMetricsFetchErr error if metricsWaitErr := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output, lastMetricsFetchErr = g.getMetricsFromPod(g.client, metricsProxyPod, metav1.NamespaceSystem, kubeSchedulerPort) return lastMetricsFetchErr == nil, nil }); metricsWaitErr != nil { err = fmt.Errorf("error waiting for scheduler pod to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr) return } }) </ins> if err != nil { return SchedulerMetrics{}, err }

! /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig instrumentation /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The flake happened in before.
This is fixed. !
/reopen This is still failing. Sample run:
Reopened this issue. <detailsIn response to : /reopen Sample run: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
The error message in makes me think the apiserver is not available during that test run: . Can someone link to the frequency graphs to see if the problem is widespread?
got it:
I don't think I was right above. I think the "the server is currently unable to handle the request" is an error coming from the proxy. The proxy was around the time this issue was filed (), so it seems likely to be related to it. can you take a look?
This is a known issue, and is working on it . Now that is being blocked by the debugging handler problem, could we merge to mitigate this problem first? cc
I would prefer to reach a decision about enabling the debug handlers, but if that's not possible, then merging PR as mitigation might make sense.
Think I hit this also.
It appears to still be failing based on
Why rebase? Any new test run will be based on merge with current k/k master and thus contain
<img width="1111" alt="Screen Shot 2021-05-28 at 13 04 52" src="https://user-" looks like it has become more stable now.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Example: apply_example, Run: func(cmd *cobra.Command, args []string) { cmdutil.CheckErr(validateArgs(cmd, args)) <del> cmdutil.CheckErr(cmdutil.ValidateOutputArgs(cmd)) </del> cmdutil.CheckErr(validatePruneAll(options.Prune, cmdutil.GetFlagBool(cmd, "all"), options.Selector)) cmdutil.CheckErr(RunApply(f, cmd, out, errOut, &options)) },

What happened: What you expected to happen: yaml output How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: Related PR


Question: return "", msg, ErrImagePullBackOff } m.logIt(ref, v1.EventTypeNormal, events.PullingImage, logPrefix, fmt.Sprintf("Pulling image %q", container.Image), klog.Info) <ins> startTime := time.Now() </ins> pullChan := make(chan pullResult) m.puller.pullImage(spec, pullSecrets, pullChan, podSandboxConfig) imagePullResult := <-pullChan

What would you like to be : The 'Successfully pulled image' event should contain the duration for image pulling. Why is this needed: Currently the success event is in this form: User has to trace back to the corresponding event for the beginning of image pulling and calculate the duration him/herself:
/sig node /assign


Question: <ins> apiVersion: apps/v1 kind: Deployment metadata: name: my-dep-null spec: selector: matchLabels: l1: l1 template: metadata: labels: l1: l1 spec: containers: - name: nginx image: registry.k8s.io/nginx:1.7.9 resources: requests: memory: "64Mi" cpu: null terminationMessagePolicy: null </ins>

As per , I'm trying to remove CPU limits. But the behaviour is different depending whether the ResourceQuota exists or not: I would expect to be idempotent, and for its behaviour to result in the same end configuration regardless of the initial configuration. a new namespace named this command: that the ResourceQuota is created and is set to it again that is removed each subsequent time it's run, as long as resourcequota/compute-resource exists resourcequota/compute-resource again that the ResourceQuota is created and is set to again, and so forth the exact same behaviour if is changed to No response <details</details<detailsOpenShift 4.13.12 </details<details</details<details</details<details</details<details</details
/sig node
Tried to replicate the issue and observed the same behaviour.
/remove-sig node /sig api-machinery /cc
What's the goal of applying here? What are you trying to achieve? I agree that this is not the right behavior though.
I'm trying to remove a previously set property. I cannot remove the ResourceQuota altogether because it contains other properties (which I left out of my example) that I do not wish to remove.
But 0 is the same as unset, isn't it?
No, if I set it to 0 then any deployment with set to higher than 0 will fail to create any pods because it will exceed the quota.
/triage accepted /cc
/assign
Hello, I'm having a similar issue. Here's my use case: We use Flux to deploy things on our cluster - in particular, we use the component and set chart values in . Some of the charts we deploy have default values we want to remove, so we want to set a value to Editing an HelmRelease with this kind of ajustments drops the null keys, so default values end up being setup again and generated manifests are invalid :( Let me know if you need anything else to keep this issue moving!
I think the correct behavior is to set the struct to the default value when or , because in the standard library does this as well, and I think we expect the result to be as similar as possible to . So our real issue here is that we didn't set the to its default value when patch.
Based on the above, I think this question is not related to the current issue.

Tracing it back to , maybe the current result is what we want. The explicit null values means that we want to remove it in patch. As mentioned , our behavior in create is also as expected. I'm not quite sure if we actually want to fix it, and how /cc I hope to get your advice as you are the implementer of
I think client-side on create should do what on update will accomplish, namely, explicit nulls in the manifest are treated as the removal of the attributes they are values of. Proof of concept opened at , I'd be happy for someone to pick that up, add test coverage, and work through any failing tests that change uncovers
Thank you, but I still have a question. If we solve the difference between create and patch behaviors in ApiServer on the client side, will third-party callers still have the same problem? Or should we not cover this scenario?
I would consider changing behavior of null in a create request server-side to be a compatibility break. values are defined for client-side apply to mean "remove the key associated with this value". I think fixing the client-side apply creation case to have the same semantic is the most narrowly scoped fix we can make while preserving compatible server-side behavior generally.
Okay, I got it, I'll follow up with


Question: func TestMonitorNodeStatusEvictPods(t *testing.T) { fakeNow := metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC) evictionTimeout := 10 * time.Minute <ins> labels := map[string]string{ kubeletapis.LabelZoneRegion:        "region1", kubeletapis.LabelZoneFailureDomain: "zone1", } </ins> // Because of the logic that prevents NC from evicting anything when all Nodes are NotReady // we need second healthy node in tests. Because of how the tests are written we need to update

<!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.1.51+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-06-20T10:51:10Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.1.51+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-06-20T11:29:14Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} What happened: After adding label to a node, node controller panics. What you expected to happen: Everything should still work. How to reproduce it (as minimally and precisely as possible): Run and wait for the node controller to panic: Anything else we need to know: I've checked the code and it looks like the node controller expects the label to be present during node registration and it can not be later. Even if the label can not be latter it shouldn't panic.
/sig node
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/assign let me try this :).


[ ] Promotion created According to this APIsnoop query, there are still some remaining ServiceAccount endpoints which are untested. Create a ServiceAccount with a static label Create a Secret Patch the ServiceAccount with a new Label and a new Secret Get the ServiceAccount to ensure it's patched List all ServiceAccounts in all Namespaces find the ServiceAccount(1) ensure that the ServiceAccount is found and is patched Delete Namespaced ServiceAccount(1) via a Collection with a LabelSelector Delete the Secret Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
Fine with moving forward You don't need to check the status of the secret as part of the test. In other places we check that the resource in question happens, we don't have to follow.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/retitle Write CoreV1ServiceAccount resource lifecycle test+promote - +3
/close close this Issue in favour of and place in In Progress Issues on org dashboard
Closing this issue. <detailsIn response to : /close close this Issue in favour of and place in In Progress Issues on org dashboard </details


Question: if err := verifyKubectlExplain(f.Namespace.Name, crd.Crd.Spec.Names.Plural+".spec", `(?s)DESCRIPTION:.*Specification of Foo.*FIELDS:.*bars.*<[]Object>.*List of Bars and their specs`); err != nil { framework.Failf("%v", err) } <del> if err := verifyKubectlExplain(f.Namespace.Name, crd.Crd.Spec.Names.Plural+".spec.bars", `(?s)FIELD:.*bars.*<[]Object>.*DESCRIPTION:.*List of Bars and their specs.*FIELDS:.*bazs.*<[]string>.*List of Bazs.*name.*<string>.*Name of Bar`); err != nil { </del> <ins> if err := verifyKubectlExplain(f.Namespace.Name, crd.Crd.Spec.Names.Plural+".spec.bars", `(?s)(FIELD|RESOURCE):.*bars.*<[]Object>.*DESCRIPTION:.*List of Bars and their specs.*FIELDS:.*bazs.*<[]string>.*List of Bazs.*name.*<string>.*Name of Bar`); err != nil { </ins> framework.Failf("%v", err) }

master-informing kubeadm-kinder-upgrade-1-26-latest kubeadm-kinder-latest 03-15 04:46 IST No response /sig api-machinery
this is a conformance test, so something has to happen with this specific environment, or it should be failing in all jobs The fact that the job name has upgrade in the name indicates that there can be some skew problem, which e2e test run this job and against which version?
Confirmed this is passing in other jobs This is related to this specific job
Can we add to the conformance test? I tried a fix like
/triage accepted


Question: currState, ok := cache.podStates[key] switch { case ok && cache.assumedPods.Has(key): <ins> // When assuming, we've already added the Pod to cache, // Just update here to make sure the Pod's status is up-to-date. if err = cache.updatePod(currState.pod, pod); err != nil { klog.ErrorS(err, "Error occurred while updating pod") } </ins> if currState.pod.Spec.NodeName != pod.Spec.NodeName { // The pod was added to a different node than it was assumed to. klog.InfoS("Pod was added to a different node than it was assumed", "podKey", key, "pod", klog.KObj(pod), "assumedNode", klog.KRef("", pod.Spec.NodeName), "currentNode", klog.KRef("", currState.pod.Spec.NodeName)) <del> if err = cache.updatePod(currState.pod, pod); err != nil { klog.ErrorS(err, "Error occurred while updating pod") } } else { delete(cache.assumedPods, key) cache.podStates[key].deadline = nil cache.podStates[key].pod = pod </del> <ins> return nil </ins> } case !ok: // Pod was expired. We should add it back.

Some Pods in Kubernetes scheduler's internal cache don't have condition in their status. We store scheduled pods information into cache as assumedPod before binding Pod. And, just after binding Pod, we receive event for scheduled Pods from eventhandler and will be used to add Pod to cache. But, as described above, we've store scheduled pods information. So, basically, scheduler doesn't update pod in cache here: But, event contains changes not only of but also of . So, we should update cache to keep up-to-date when we receive event for scheduled Pods from eventhandler. Otherwise, won't updated until new changes for that Pod are made. Kubernetes scheduler's internal cache always keeps Pod's up-to-date. some Node. a Pod. is scheduled by scheduler. in Pods in Kubernetes scheduler's internal cache (by adding debug log or cache) I think kubelet will update Pods status after they run containers, so you need to do (4) before kubelet does so. (Or doesn't use kubelet like e2e tests) /assign Let me double-check this. (The above explanation includes my expectations, so may not be entirely correct.) v1.23.4 But, I guess this bug exists on latest master as well. <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
/remove-lifecycle stale
Does this lead to unschedulable pods or is this just for debugging purposes?
We uses condition in a custom plugin. Mostly No in the real cluster. Only for a very short time (from the scheduler schedules a Pod to kubelet next updates a pod) our plugin does not behave as expected and may make some pods unschedulable. But, when it comes to e2e testing, this bug can be troublesome. In our e2e testing, we don't run kubelet, and no one updates Pods after the scheduler scheduled them Then, all scheduled Pods in the scheduler cache have non-updated condition and it results in unexpected results from our plugin.


Question: import ( "fmt" <ins> "sort" </ins> "time" "k8s.io/kubernetes/pkg/api"

I've noticed at least 10 separate test flake issues open in which at least one failure looks like this: Current list: &q=is%3Aissue%20label%3Akind%2Fflake%20is%3Aopen%20%22was%20not%20deleted%20within%20limit%22 The 10 current issues (note that some are 'broken test run' issues and might have multiple failure causes): cc
Just to clarify, the pods were deleted in the test (hence the message). I mentioned this before in
I was chasing down another p0 but now that that is resolved will try to look for more that we can do here. On Wednesday, July 27, 2016, Yu-Ju Hong wrote:
- why did you close this - this is happening quite frquently.
Thanks for reopening! We will now automatically choose and owner for auto-opened flake issues. I closed a bunch of old flakes that were created with no assignee. I want them to get recreated with an assignee. The hope is someone will actually look at these.
is another example of this flake.
has the same symptom too.
One more example: would it help if we bump the timeout from 5 min to N (e.g., 8) min? or should we ask tests that create significant number of pods to clean up after themselves?
I am going to try and look at optimizing the requeue behavior for tests that have a lot of pods and we are waiting for them to delete. Looks like once they delete, we don't observe their deletion until the next resync
Opened a PR to improve e2e framework ns deletion to avoid duplicate ns delete operations, and to give more detailed information on what content did not actually get deleted.


Question: return true, nil } <ins> // check if a Pod is controlled by a Replication Controller in the List func hasReplicationControllersForPod(rcs *api.ReplicationControllerList, pod api.Pod) bool { for _, rc := range rcs.Items { selector := labels.SelectorFromSet(rc.Spec.Selector) if selector.Matches(labels.Set(pod.ObjectMeta.Labels)) { return true } } return false } </ins> // waitForPodsRunningReady waits up to timeout to ensure that all pods in <del> // namespace ns are running and ready, requiring that it finds at least minPods. // It has separate behavior from other 'wait for' pods functions in that it re- // queries the list of pods on every iteration. This is useful, for example, in // cluster startup, because the number of pods increases while waiting. </del> <ins> // namespace ns are either running and ready, or failed but controlled by a // replication controller. Also, it ensures that at least minPods are running // and ready. It has separate behavior from other 'wait for' pods functions in // that it requires the list of pods on every iteration. This is useful, for // example, in cluster startup, because the number of pods increases while // waiting. </ins> func waitForPodsRunningReady(ns string, minPods int, timeout time.Duration) error { c, err := loadClient() if err != nil {

On a healthy test cluster the health check looks like this: On a failed cluster it looks like this:
Possibly related to ?
this has elasticsearch logging failing.
As per discussion with I think that the solution here is to ignore failed pods for these test cluster health checks. The failed pods were all correctly restarted by the relevant replication controllers, so the tests should have been able to proceed in this and all the other similar cases that we looked at. Why the pods failed is a separate question, which will hopefully be detected and fixed elsewhere.
It might be due to
Just waiting for oncall to merge


Question: // restart iptables"?). So instead we just manually delete all "KUBE-" // chains. <del> ginkgo.By("dumping iptables rules on a node") </del> <ins> ginkgo.By("dumping iptables rules on node " + host) </ins> result, err := e2essh.SSH("sudo iptables-save", host, framework.TestContext.Provider) <ins> e2essh.LogResult(result) </ins> if err != nil || result.Code != 0 { <del> e2essh.LogResult(result) </del> framework.Failf("couldn't dump iptable rules: %v", err) }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-features Which test(s) are failing: Networking should recreate its iptables rules if they are deleted [Disruptive] Since when has it been failing: Commit Testgrid link: Reason for failure: Anything else we need to know: Possibly related to this? It's listed in the git diff /sig network
/assign /cc Thanks for detecting it so quicklly
most probably is related to
/unassign /assign he has already submitted a fix
/triage accepted
Reviewed. Marking as resolved on the CI Signal Board. Once was merged, failures stopped and they have not re-occurred in the last 8 days (Oct 10th)
Yay


Question: } mapper, typer := f.Object() <del> b := resource.NewBuilder(mapper, typer, f.ClientMapperForCommand(cmd)). </del> <ins> b := resource.NewBuilder(mapper, typer, f.ClientMapperForCommand()). </ins> ContinueOnError(). NamespaceParam(cmdNamespace).DefaultNamespace(). SelectorParam(selector).

I think the need for this argument to the factory methods has passed - it means that callers who want to provide configurable behavior have to do it through the factory itself, which is acceptable. We can remove it whenever.
I removed most of the unused arguments. But, There are a few methods still using the argument. Namely, , , and . Before proceeding any further I wanted to ask what should be done about these methods? Deprecated/Refactored/Removed ?
The factory methods were the most important. The pattern before was people using the flags by name (cmd.Flags().Get("foo")), but now factory methods must use pointer flag vars, so there's nothing that should need the command there. The printer methods are optional units that can be used when you add the flags for them to your command. Unless we refactor the definition of those flags to use pointers and return a printer struct (something I had started a while back and never finished), it's not worth changing them right now. The other consumer of factory is OpenShift - we wrap the factory to add our additional types (implementations of things like describers, action verbs, etc): - but we will pick up and remove use of cmd when we rebase.
Thanks for the explanation. I've cleaned the as well. Print method APIs are untouched as you suggested.
cc
Looks like this was done by PR


Question: return fmt.Sprintf("%d-%d", pr.Base, pr.Base+pr.Size-1) } <del> // Set parses a string of the form "min-max", inclusive at both ends, and </del> <ins> // Set parses a string of the form "value", "min-max", or "min+offset", inclusive at both ends, and </ins> // sets the PortRange from it.  This is part of the flag.Value and pflag.Value // interfaces. func (pr *PortRange) Set(value string) error { <del> value = strings.TrimSpace(value) </del> <ins> const ( SinglePortNotation = 1 << iota HyphenNotation PlusNotation ) </ins> <del> // TODO: Accept "80" syntax // TODO: Accept "80+8" syntax </del> <ins> value = strings.TrimSpace(value) hyphenIndex := strings.Index(value, "-") plusIndex := strings.Index(value, "+") </ins> if value == "" { pr.Base = 0

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Implement this TODO: What you expected to happen: Start kube-proxy with a 'beginPort+offset' format port range configuration. This is needed when I want to allocate specific numbers of port from a host without extra manual calculation. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
/sig network Does anyone besides kube-proxy use this?
/assign


Question: srcs = [ "active_deadline.go", "doc.go", <ins> "errors.go", </ins> "kubelet.go", "kubelet_getters.go", "kubelet_network.go",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pods are synchronized by kubelet before network is ready. Synchronization in such case fails and pod is synchronized after 10 seconds. What you expected to happen: Kubelet should wait until network is ready before starting synchronizing pods. How to reproduce it (as minimally and precisely as possible): Configure cluster with Cluster Autoscaler enabled. Start pod so cluster is scaled up. In logs you might find following error: E0907 13:18:49. 972 ] Error syncing pod xxxxxx ("xxxxxxxxx(xxxxxxxxxxx)"), skipping: network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] Environment: Build from HEAD Cloud provider or hardware configuration: GKE
/kind bug /sig node


Question: <ins> # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. FROM alpine:3.5 RUN apk add --no-cache redis COPY redis.conf /etc/redis.conf EXPOSE 6379 VOLUME /data CMD ["redis-server", "/etc/redis.conf"] </ins>

Kubernetes version (use ): master What happened: Pulling the redis image takes a while, like 30 seconds in GCE. This might be leading to periodic timeouts in tests like . What you expected to happen: Tests shouldn't depend on large images that take a potentially long time to pull. proposed fix to reduce flakiness we can increase the replica count on this test to 2 or make the redis image we use smaller (or not use redis at all and just use a simpler smaller image)
cc (considering this as a bug since we shouldnt need a 500MG image just to test kubectl :))
This should be as simple as changing the current to use alpine and install redis, instead of basing it off debian, installing gcc, and compiling redis from source :-) The image should be ~10MB.


Question: if selector.Empty() || !selector.Matches(labels.Set(pod.Labels)) { continue } <ins> // Existing in DisruptedPods means it has been processed in API server, // we don't treat it as a violating case. if _, exist := pdb.Status.DisruptedPods[pod.Name]; exist { continue } // Only decrement the matched pdb when it's not in its <DisruptedPods>; // otherwise we may over-decrement the budget number. pdbsAllowed[i]-- </ins> // We have found a matching PDB. <del> if pdbsAllowed[i] <= 0 { </del> <ins> if pdbsAllowed[i] < 0 { </ins> pdbForPodIsViolated = true <del> } else { // Only decrement the matched pdb when it's not in its <DisruptedPods>; // otherwise we may over-decrement the budget number. if _, exist := pdb.Status.DisruptedPods[pod.Name]; !exist { pdbsAllowed[i]-- } </del> } } }

In one recent fix (), we enhanced the preemption logic to respect . But there is a glitch: if we changed the DisruptedPods in L1499 from "a" to "b", the test would fail it would also fail if we change the pods (i.e., []Pod{"b", "a"}) ordering in L1496~L1497 This is a problem we need to fix. On the other hand, I'm thinking about the real semantics of . According to the docs, if a pod is matched by the "DisruptedPods" list, it denotes the pod is about to be evicted by API server as it at least has violated one disruption budget. Given this fact, IMO in scheduler preemption, we should treat this kind of pod as "nonViolating" in all cases - no matter the current is negative or not. So that the scheduler would treat them as preemption candidates. /sig scheduling
/assign
cc/


Question: for _, endpointSlice := range slicesToCreate { addTriggerTimeAnnotation(endpointSlice, triggerTime) <del> _, err := r.client.DiscoveryV1beta1().EndpointSlices(service.Namespace).Create(context.TODO(), endpointSlice, metav1.CreateOptions{}) </del> <ins> createdSlice, err := r.client.DiscoveryV1beta1().EndpointSlices(service.Namespace).Create(context.TODO(), endpointSlice, metav1.CreateOptions{}) </ins> if err != nil { // If the namespace is terminating, creates will continue to fail. Simply drop the item. if errors.HasStatusCause(err, corev1.NamespaceTerminatingCause) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With EndpointSlice feature enabled, after EndpointSliceController creates or updates an EndpointSlice when reconciling a Service, the Service would always be processed one extra time, which is triggered by the EndpointSlice /updated event made by itself during the first processing. What you expected to happen: EndpointSliceController shouldn't reprocess a Service due to the EndpointSlice update made by itself. How to reproduce it (as minimally and precisely as possible): EndpointSlice feature, improve kube-controller-manager log to 5. a nginx deployment a Service backed by the above deployment. kube-controller-manager log: the service will be processed twice in succession. Anything else we need to know?: It's because EndpointSliceController tracks stale resource version after creating/updating an EndpointSlice. Environment: Kubernetes version (use ): v1.18.0-alpha.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: return } <ins> const containerNamePrefix = "k8s" </ins> // Creates a name which can be reversed to identify both manifest id and container name. func manifestAndContainerToDockerName(manifest *api.ContainerManifest, container *api.Container) string { // Note, manifest.Id could be blank. <del> return fmt.Sprintf("%s--%s--%x", escapeDash(container.Name), escapeDash(manifest.Id), rand.Uint32()) </del> <ins> return fmt.Sprintf("%s--%s--%s--%x", containerNamePrefix, escapeDash(container.Name), escapeDash(manifest.Id), rand.Uint32()) </ins> } // Upacks a container name, returning the manifest id and container name we would have used to

release-master-blocking Conformance-GCE-master-kubetest2Kubernetes e2e suite.[It] [sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance] ci-kubernetes-gce-conformance-latest-kubetest2.Overall kubetest2.Test 07-11 15:51 EEST No response /sig cloud-provider cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I remember some timeout has changed with ginkgo2 and something moved from 24hours to 1 hours, or something like that, is it possible that this is the problem here, in this job?
The Grinko timeout change could also be the root problem with this job:
hmm, we have to revisit the flags
hmm, this seems correct
that was a good observations, seems that job is failing because gathering the logs takes too much time !
there is no timeout set in the kubetest2 command here (not sure if this would be the place to set a timeout)
it seems kubetest has to be migrated to ginkgov2 it should use this as basis I'm on vacation this week, can somebody take over and open the issue and patch kubetest2?
looks like this file needs to get updated: add line 72:
oh, you already commented this :D
I will have a look later
Related: umbrella issue:
last testgrid run passed :)


Question: if o.Resource != (schema.GroupVersionResource{}) || o.ResourceName != "" { return fmt.Errorf("NonResourceURL and ResourceName can not specified together") } <ins> } else if !o.Resource.Empty() && !o.AllNamespaces && o.DiscoveryClient != nil { if namespaced, err := isNamespaced(o.Resource, o.DiscoveryClient); err == nil && !namespaced { if len(o.Resource.Group) == 0 { fmt.Fprintf(o.ErrOut, "Warning: resource '%s' is not namespace scopedn", o.Resource.Resource) } else { fmt.Fprintf(o.ErrOut, "Warning: resource '%s' is not namespace scoped in group '%s'n", o.Resource.Resource, o.Resource.Group) } } </ins> } if o.NoHeaders {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What you expected to happen: if returns true I expect to be able to perform the action without issue. Note in this case my service user has a rolebinding in a single namespace (and that is also what my kubecontext is set to.) if I change my default to a different namespace then the command returns false How to reproduce it (as minimally and precisely as possible): create a namespace (test) create a service account in that namespace create a role that gives get list and watch permissions to create a rolebinding from that role to the service account. Anything else we need to know?: this is special because namespaces are not scoped to a namespace. if you replace the command with then you get the exepcted result however this breaks the path of least suprise. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig auth
asks the authorization system if the given verb can be performed on the given noun at the given scope. It does not interpret whether the resource exists at that scope. We could warn if we find the specified resource and it does not exist at the specified scope (just like we warn if you ask about a resource that we can't find), but we would still interrogate the authorization system and return the results as we do today. /kind feature /remove-kind bug
/cc


Question: nodePortAddressStrings []string, kernelHandler KernelHandler, ) (*Proxier, error) { <del> // Proxy needs br_netfilter and bridge-nf-call-iptables=1 when containers // are connected to a Linux bridge (but not SDN bridges).  Until most // plugins handle this, log when config is missing if val, err := sysctl.GetSysctl(sysctlBridgeCallIPTables); err == nil && val != 1 { klog.InfoS("Missing br-netfilter module or unset sysctl br-nf-call-iptables, proxy may not work as intended") } </del> // Set the conntrack sysctl we need for if err := proxyutil.EnsureSysctl(sysctl, sysctlVSConnTrack, 1); err != nil { return nil, err

The check in: is incorrect. It misses the case when the sysctl doesn't exist (err != nil). /sig network /area ipvs /area kube-proxy The check sould be: To reproduce you must use a kernel without the function, or the module unloaded. This is likely not easy if a regular Linux distro is used. But IMO no e2e testing should be needed (none exist now). It's just a warning that may be missed, so nothing urgent. master (all) N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue </details
removed one of those , it looks that we should remove this to be consistent
I can work on removing the code to maintain consistency. Can you please assign this issue to me
/assign You can also use to assign the issue to yourself.
oops, I should have thought to check ipvs backend too


Question: // On gci, root is read-only and controller-manager containerized. Assume // controller-manager has started with --flex-volume-plugin-dir equal to this // (see cluster/gce/config-test.sh) <del> gciVolumePluginDir        = "/home/kubernetes/flexvolume" gciVolumePluginDirLegacy  = "/etc/srv/kubernetes/kubelet-plugins/volume/exec" gciVolumePluginDirVersion = "1.10.0" detachTimeout             = 10 * time.Second </del> <ins> gciVolumePluginDir = "/home/kubernetes/flexvolume" detachTimeout      = 10 * time.Second </ins> ) // testFlexVolume tests that a client pod using a given flexvolume driver

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I think these jobs do not have the flexvolume directory properly configured. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign


Question: firstServiceSelector = getFirstServiceSelector(pod, s.serviceLister) } //pods matched namespace,selector on current node <del> matchedPodsOfNode := filteredPod(pod.Namespace, firstServiceSelector, nodeInfo) </del> <ins> var selectors []labels.Selector if firstServiceSelector != nil { selectors = append(selectors, firstServiceSelector) } score := countMatchingPods(pod.Namespace, selectors, nodeInfo) </ins> return schedulerapi.HostPriority{ Host:  node.Name, <del> Score: int(len(matchedPodsOfNode)), </del> <ins> Score: score, </ins> }, nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: /kind bug What you expected to happen: Problem 1: Predicates filter nodes. Existing pods on those nodes will be not be counted when calculating max pods per (zone or node) resulting in imbalanced cluster. Eg: removes nodes which cannot . If any of the pods are already scheduled on this node, they are not considered when counting in . Problem 2: When there are 2 selectors(service and replication controller), it is sufficient to match any one selector for distribution. This creates imbalance ]. Pods from previous deploys matches and are counted when distributing pods across zones/nodes (Even though they do not match ) . These pods will be deleted. After the deploy completes, the cluster is imbalanced - by zone and/or pods per node. How to reproduce it (as minimally and precisely as possible): Problem 1: Get pods to get scheduled on nodes with very high cpu utilization [Existing cpu utilization + 1 new pod will result in almost equal to 0]. GeneralPredicate will drop this node after scheduling the first pod. This will lead to over utilization of an already loaded zone. If there are enough nodes like this one, it will create a pile on effect resulting in most pods scheduled in an already loaded zone. Problem 2: Have 2 selectors and deploy 30 pods across 3 zones. Twice. Anything else we need to know?: Environment: Kubernetes version (use ): /sig scheduling <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
re Problem 1: for per node, we only need to compare current nodes as they will share the same maxPods. for per zone, we need to account other nodes, e.g. because of resource predicates. re problem 2: a little about your case :(
Problem 1: : It does not matter if the denominator is all nodes or current nodes. One with higher utilization will still end up with a lower score. : It is critical that the denominator is all nodes. Else we are ending up with highly imbalanced cluster. Problem 2: Here is an example list of selectors.

All pods matching any one service selector are counted in So in our case, old pods are counted during scheduling. But they get killed during the deployment. If the maxSurge and maxUnavailable is high enough, you will end up with an imbalanced zone after deployment.
Gentle ping.
/assign bsalamat
Re: Problem 1 - You can also reproduce this by using pod anti-affinity. Zone score is calculated only based on nodes filtered by predicates. IMO zone score should be calculated using all nodes.
I will try to separate this into two PRs. Any other suggestions welcome.
I am working Problem 1. Looks like I have to open a new issue.


Question: client := NewClient(u, c.k) client.Namespace = "urn:" + namespace <ins> client.Transport.(*http.Transport).TLSClientConfig = c.Transport.(*http.Transport).TLSClientConfig </ins> if cert := c.Certificate(); cert != nil { client.SetCertificate(*cert) }

Expectation- We should be able to create PVCs via a storageclass with storagePolicyName defined while using a singed ca cert for secure communications with vCenter. Bug- The PVC created with a storageclass with storagePolicyName defined fails with a error when PVCs created with a storageclass without the storagPolicyName work against the same vCenter and datastore. Scenario: Utilizing the vSphere in-tree plugin on kubernetes 1.12.7. Utilizing secure communication with vSphere (signed ca cert imported into the cluster) We are able to create PVCs with the following storage class: However, if we include a storagePolicyName we receive a failure: Here is the yaml for our sc1 storage class: /sig vmware
I think we are having issue with creating PBM client using certificate set on the vCenter Client. We need to create a PBM client to get list of compatible datastore for given storage policy. I think call is failing at - Can you please provide controller-manager logs for further debugging. Make sure log level is set to 4.
Sorry for the delayed response. I'm not confident I got the log level correct, however this is what I find:
I believe I have log level 4 now set:
For comparison, here is a PVC created w/o storagePolicyName on the same datastore which succeeds:
thanks for posting logs. Now this is confirmed, issue is happening while creating PBM client when we use certificate based authentication. One more thing I want to confirm regarding the certificate, are you using solution user as mentioned here -
Can you also provide value of in the file? Is it set to true or false / 1 or 0 ? Also confirm version of vCenter you are using.
is unset, as for the solution user I'd have to leave that to to expand on.
I don't believe we used any solution user because we simply used "" as the default user during installation. please correct me if I'm wrong.
How is the certificate and private key retrieved for the user . I am successfully able to connect to the PBM using vpxd certificte and key. I ssh into the vCenter and executed following commands to retrieve certificate and private key for vpxd. with vpxd cert and private key, I do not see this issue. In addition I also set while making connection.

Can you try setting Insecure to false? That's the flag setting that triggers the cert error.
with , I see the exact same error - , while creating VC client.
Isn't that because it's self-signed? In the case of the issue in this ticket the cert is signed by a trusted CA /cc to confirm.
The cert is self-signed and its CA isn't trusted.
I used vpxd certificate and issuer is I require to set .
Is there any way to make it work if insecure=false?
- then I’d say that’s expected? If the root CA isn’t trusted, then it is insecure?
well, while that's expected, I don't know if there's a method to add the root CA to a trusted list.
Correct me if I’m wrong but isn’t the trusted list the root CA store of the K8S node’s OS in this case? I’d imagine whatever CA Root signed the cert would need to be to the OS of choice’s CA store.
you are correct, we need to download trusted root certificate and add them in the store on node vm. For now on my mac, I have downloaded trusted root certificate from the vCenter -and them in the keychain and flagged as "Always Trust". ! After this, I am able to connect to VC client with or without specifying this flag at all using VPXD certificate and private key. So certificate needs to be to the CA trust store on the master node VM. after adding trusted root certificate from vCenter to the trust store, you should be able to make a connection using .
After importing the root cert by placing it in /etc/ssl/certs on the master node, I'm able to provision a PVC while specifying a storage policy. Thanks for all the help.
Can you close this issue and work with KK to get this automated in the product?
It turns out that importing the root cert into the host was just a workaround, and the actual bug was fixed in That fix is included in govmomi v0.21.0.


Question: <del> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package filters import ( "net/http" "runtime/debug" "github.com/golang/glog" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/util/runtime" apirequest "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/server/httplog" ) // WithPanicRecovery wraps an http Handler to recover and log panics. func WithPanicRecovery(handler http.Handler, requestContextMapper apirequest.RequestContextMapper) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { defer runtime.HandleCrash(func(err interface{}) { http.Error(w, "This request caused apisever to panic. Look in log for details.", http.StatusInternalServerError) glog.Errorf("APIServer panic'd on %v %v: %vn%sn", req.Method, req.RequestURI, err, debug.Stack()) }) logger := httplog.NewLogged(req, &w) var requestInfo *apirequest.RequestInfo ctx, ok := requestContextMapper.Get(req) if !ok { glog.Errorf("no context found for request, handler chain must be wrong") } else { requestInfo, ok = apirequest.RequestInfoFrom(ctx) if !ok { glog.Errorf("no RequestInfo found in context, handler chain must be wrong") } } if !ok || requestInfo.Verb != "proxy" { logger.StacktraceWhen( httplog.StatusIsNot( http.StatusOK, http.StatusCreated, http.StatusAccepted, http.StatusBadRequest, http.StatusMovedPermanently, http.StatusTemporaryRedirect, http.StatusConflict, http.StatusNotFound, http.StatusUnauthorized, http.StatusForbidden, http.StatusNotModified, apierrors.StatusUnprocessableEntity, http.StatusSwitchingProtocols, ), ) } defer logger.Log() // Dispatch to the internal handler handler.ServeHTTP(w, req) }) } </del>

This is simple cleanup. It is unnecessarily disconcerting to read through apiserver logs and find every log line is from the file "". We should adjust the code printing these lines, either by changing the file name, or (ideally) by making it more intelligent about logging the source of the call (e.g., callers--).
, i'd like to try this one :).
No activity on this one, will try to fix it.
Has it been resolved? Can I try?
check out for discussion about it.


Question: } // Keep trying to create an invalid instance of the CRD until we <del> // get an error containing the ResourceVersion we are looking for </del> <ins> // get an error containing the message we are looking for </ins> // counter := 0 return wait.PollUntilContextCancel(context.TODO(), 100*time.Millisecond, true, func(_ context.Context) (done bool, err error) {

integration TestRatchetingFunctionality/Enum TestRatchetingFunctionality/MinPropertiesMaxProperties TestRatchetingFunctionality/CELOptionalOldSelf TestRatchetingFunctionality/ TestRatchetingFunctionality/ since 3/8 No response /sig api-machinery
/assign /triage accepted


Question: - [Urgent Upgrade Notes](#urgent-upgrade-notes) - [(No, really, you MUST read this before you upgrade)](#no-really-you-must-read-this-before-you-upgrade) - [Known Issues](#known-issues) <del> - [Etcd v3.5.[0-2] data corruption](#etcd-v350-2-data-corruption) </del> <ins> - [Etcd v3.5.[0-2] data corruption](#etcd-v350-2-data-corruption) </ins> - [Changes by Kind](#changes-by-kind-14) - [Deprecation](#deprecation) - [API Change](#api-change-5)

The section of the Changelog for 1.23.14 (and sibling bugfix releases) says that 1.23.13 both has the vuln and also has the fix. The Fix version should be 1.23.14 in the above and sibling changelogs. Load the link above. No response N/A N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig docs
cc
For v1.23.14, the title is . This seems to be valid. means
You're looking at the wrong text. Follow the link and see .13 in both lists: kube-apiserver v1.25.0 - v1.25.3 kube-apiserver v1.24.0 - v1.24.7 kube-apiserver v1.23.0 - v1.23.13 kube-apiserver v1.22.0 - v1.22.15 kube-apiserver <= v1.21.? kube-apiserver v1.25.4 kube-apiserver v1.24.8 kube-apiserver v1.23.13 kube-apiserver v1.22.16
Got it. Thanks for pointing out the location.
I opened to fix it.
/reopen Can you please cherry-pick to the release branches (1.25-1.22)?
Reopened this issue. <detailsIn response to : /reopen Can you please cherry-pick to the release branches (1.25-1.22)? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Of course. Done
Cherry-picks are done, thanks a lot /close
Closing this issue. <detailsIn response to : Cherry-picks are done, thanks a lot /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } r := &Reflector{ <del> name:                   "test-reflector", listerWatcher:          lw, store:                  NewFIFO(MetaNamespaceKeyFunc), initConnBackoffManager: bm, clock:                  fakeClock, watchErrorHandler:      WatchErrorHandler(DefaultWatchErrorHandler), </del> <ins> name:              "test-reflector", listerWatcher:     lw, store:             NewFIFO(MetaNamespaceKeyFunc), backoffManager:    bm, clock:             fakeClock, watchErrorHandler: WatchErrorHandler(DefaultWatchErrorHandler), </ins> } r.MaxInternalErrorRetryDuration = tc.maxInternalDuration

Currently, reflector uses two backoff managers: One is used for initial list call and other for watch call. Imagine following scenaraion when k8s apiserver is under very heavy load: the beginning, reflector spams k8s apiserver (with exponential backoff) with calls until one of them succeeds (due to APF throtling and receiving ) tries to run request and uses separate backoffManager - so again spams k8s apiserver with requests Watch does not succeed during Etcd compaction window (2.5 - 5 minutes usually) at some point it will receive and go back to point 1, spamming k8s apiserver again with List calls (because first backoffManager for List got reseted after 2 minutes of retries for Watch requests) I propose we combine these two backoff managers into one. If we combine these two backoffManagers into one, it will work like this: At the beginning, reflector spams k8s apiserver (still with exponential backoff) with calls until one of them succeeds Tries to make first watch call instantly (as there is no backoff on this path) If it fails, it will makre retries every (30, 60)s If watch request fails all the times during Etcd compaction window it will go back to List call, but already being throttled with (30, 60)s window. /sig scalability
/cc
/assign
I think the proposed solution makes sense to me. Probably before we start, we should to understand why we ended up with two back off managers by looking into PRs touching them. Maybe the code authors had some specific case in mind.
for thoughts too I agree it makes sense, but I don't know the historical context here.
/sig api-machinery
/triage accepted


Question: local wait=${3:-1} local times=${4:-30} local maxtime=${5:-1} <del> local extra_args=${6:-} </del> command -v curl >/dev/null || { kube::log::usage "curl must be installed"

What happened: After running ./hack/update-openapi- with the latest code, api/openapi- file became empty even when no changes were made in the code. What you expected to happen: After running ./hack/update-openapi- with the latest code, api/openapi- file should not be changed when no changes were made in the code. When I tried the same thing a few week back, I didn't encounter this problem. How to reproduce it (as minimally and precisely as possible): Download the latest kubernetes/kubernetes code and run the following script to update openapi spec. api/openapi- became empty after this. Anything else we need to know?: There's a line in update-openapi- that runs a curl command . This line was changed a few days ago by this commit: PR: I checked out the commit before and ran /update-openapi- It works. is not empty any more. Is there any new setting required before running ./hack/update-openapi- now? Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
cc
Hi, thanks for the report, but it is a bit weird that I could not reproduce locally: and besides, completes successfully on my machine. Could you reproduce?
Someone else also ran into this same problem and asked a question on sig-api-machinery slack yesterday. So I'm not the only one who ran into this.
could you change the args passed to curl in to and rerun the script to see if there is any error?
since it is already late here(UTC+8), I would check this issue again tomorrow
I was the "someone else" who ran into this problem :) I changed the script to use The result: Actually, it was 403 without the "S", too, but that is visible only if curl is executed in verbose mode.
Hi, thanks for your information. Given that the apiserver returned 403, I suspect the flag passed to curl has no effect in your environment, would you mind posting the output of ? I am using curl 7.64.1: In addition, I think you could try changing the curl command in to the following: and see if the problem is solved.

I tried the following and it worked for me. is no longer empty. curl version:
Thanks for your confirmation, and I have filed to address this issue.
I can confirm, it works with the changed curl command.
/triage accepted


Question: COPY --from=servercore-helper /Windows/System32/netapi32.dll /Windows/System32/netapi32.dll COPY --from=prep /coredns.exe /coredns.exe COPY --from=prep /iperf /iperf <ins> COPY --from=prep /wincoreutils/sync.exe /bin/sync.exe </ins> # NOTE(claudiub): docker buildx sets the PATH env variable to a Linux-like PATH, which is not desirable. ENV PATH="C:dig;C:bin;C:curl;C:Windowssystem32;C:Windows;C:Program FilesPowerShell;"

We'd like to add to the windows nodes Discussed in , some storage e2e tests need the call to flush data to disk, this utility is not available in windows nodes or agnhost /sig storage /sig testing
/sig testing
/sig storage /kind flake
/sig windows
Does sync only need to be in Agnhost or also on the Windows node? I don't think there is any issue with adding to the agnhost for this tests. Has it been tested that it works? /cc in case you have thoughts
In the README of sync it says: Initially I thought it could be to agnhost however told me that Windows containers cannot run in privileged mode as of today. cc
Since we run the Windows Test containers as they should have full access to modify the host files if the host path is mapped in.
I downloaded the zip file and I checked that it has an EULA, do you know which sig would help with the license?
Hm, if the sysinternal is problematic, what about the GNU one? It seems to come from coreutils, which can easily be downloaded and extracted from. Hopefully, they are 64-bit binaries. Anyways, I think adding is a good idea, there are a few other non-conformance tests that are calling sync.
I believe that would be sig-contributor-experience. I thought we were using other sys-internal tools as well but it is a good idea to double check.
/triage accepted /assign
/remove-sig testing we don't own the nodes (?)
I've sent , which would add using that PR. Let us know if that satisfies your requirements, we could merge afterwards.
Thanks I guess we can merge the PR and I can run the storage e2e tests that need .


Question: // When feature-gate for TaintBasedEvictions=true flag is enabled, // TaintNodeNotReady would be automatically added by node controller // when node is not ready, and removed when node becomes ready. <del> TaintNodeNotReady = "node.alpha.kubernetes.io/notReady" </del> <ins> TaintNodeNotReady = "node.kubernetes.io/not-ready" // DeprecatedTaintNodeNotReady is the deprecated version of TaintNodeNotReady. // It is deprecated since 1.9 DeprecatedTaintNodeNotReady = "node.alpha.kubernetes.io/notReady" </ins> // When feature-gate for TaintBasedEvictions=true flag is enabled, // TaintNodeUnreachable would be automatically added by node controller

Almost all style strings use dashes, rather than javascript names. vs .
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Thanks for notification, a quick fix is submitted.
/cc
The PR here isn't merged yet. Can we make that happen please? We're still in alpha, so should be ok
Should I add the v1.8 milestone to this issue?
Hope so. We are pending in , need approve from
I opened with safe changes. "not-ready" change needs to be discussed more, as it's a breaking change. I believe it's best with fixing that to the time we'll be moving to beta, as we'll need to do another breaking change, i.e. remove 'alpha' from the key.
Aren't we doing this at the 11th hour? Is this going to hit 1.8?
Strings like this that are part of the core API should be part of a design doc that is formally reviewed by an API group. We need to find ways to catch this stuff earlier. We should also have an evolving API style guide that lists stuff like this. /cc
Thanks , let me make some summary: quick fix for 1.8 is , will be merged soon. final fix is , we are collaborating to reach a final solution and it's not target 1.8
Agree with Joe, can someone open a PR to that documents that taint names should follow the naming conventions for annotations?


Question: return true, binding, nil }) controllers := make(map[string]string) <del> stopFn := broadcaster.StartEventWatcher(func(obj runtime.Object) { </del> <ins> stopFn, err := broadcaster.StartEventWatcher(func(obj runtime.Object) { </ins> e, ok := obj.(*eventsv1.Event) if !ok || e.Reason != "Scheduled" { return

Follow-up cleanup work for: Most of the information about this issue can be found in , where the work merged with a to change the function signature of to return a func() as well as an error. This issue will help to track that. All TODOs introduced in are resolved. N/A N/A <detailsN/A </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority low Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /triage accepted
/priority important-longterm
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: Expect(err).NotTo(HaveOccurred()) }) <del> It("Pods should be scheduled to low resource use rate node", func() { //Make sure all the schedulable nodes are balanced (have the same cpu/mem usage ratio) By("Create pods on each node except the last one to raise cpu and memory usage to the same high level") var expectedNodeName string expectedNodeName = nodeList.Items[len(nodeList.Items)-1].Name nodes := nodeList.Items[:len(nodeList.Items)-1] // make the nodes except last have cpu,mem usage to 90% createBalancedPodForNodes(f, cs, ns, nodes, podRequestedResource, 0.9) By("Create a pod,pod should schedule to the least requested nodes") createPausePod(f, pausePodConfig{ Name:      "priority-least-requested", Labels:    map[string]string{"name": "priority-least-requested"}, Resources: podRequestedResource, }) By("Wait for all the pods are running") err := f.WaitForPodRunning("priority-least-requested") framework.ExpectNoError(err) By("Verify the pod is scheduled to the expected node") testPod, err := cs.CoreV1().Pods(ns).Get("priority-least-requested", metav1.GetOptions{}) framework.ExpectNoError(err) Expect(testPod.Spec.NodeName).Should(Equal(expectedNodeName)) }) </del> It("Pods created by ReplicationController should spread to different node", func() { By("Create a pod for each node to make the nodes have almost same cpu/mem usage ratio")

Includes links to these: SchedulerPriorities [Serial] Pods should be scheduled to low resource use rate node SchedulerPriorities [Serial] Pod should be prefer scheduled to node that satisify the NodeAffinity SchedulerPriorities /sig scheduling /assign See also
we need to triage this issue to decide if there are release blocking or not.
This one would be better if you have cycles.
sure, let me try this :).
Among the tests in the first post of this issue, "Pods should be scheduled to low resource use rate node" is failing a lot. As I had mentioned in a different issue, I tried reproducing this locally, but I was not able to. So, my guess is that some configuration/condition/state in our test clusters cause the test to fail. The "least requested" priority function is tested extensively in our unit tests: So, given the issues caused by the test and dependency of scheduling decisions on the config and state of the cluster, I think we can go ahead and delete the e2e test.
Thanks ! Would it help if i broke out the other two failures into separate issues?
Please note that the other two are not failing that frequently. So, I don't see those as blockers for 1.7: https://k8s- That said, as has mentioned in , the other e2e scheduler priority tests should probably be removed eventually. Our e2e test framework is not deterministic enough to run these tests, but given that the other two are passing at a reasonable rate, I would wait until we discuss their removal in our next sig-scheduling meeting.
OK, we're going to ignore 1.5-1.7. Please focus now only on the 1.6 -1.7 tests. PTAL at this . If the view is these test failures aren't "real" failures, but are artifacts of race-conditions in cluster testing, then please delete the tests (since they give us no signal) and devote time to eliminating those race conditions so we can get reliable test coverage.
is also similar to . It is failing a lot in our e2e tests, but have been running the test in my own GCE cluster and has been passing for 18 times so far. I think this one should be deleted as well for similar reasons.


Question: "//vendor/github.com/coreos/etcd/client:go_default_library", "//vendor/github.com/coreos/etcd/clientv3:go_default_library", "//vendor/github.com/coreos/etcd/pkg/transport:go_default_library", <ins> "//vendor/github.com/grpc-ecosystem/go-grpc-prometheus:go_default_library", "//vendor/google.golang.org/grpc:go_default_library", </ins> ], )

/kind feature Can we provide a way to enable etcd clientv3 prometheus metrics at the apiserver side ? The etcd.clientv3 interface is instantiated and it does not enable metrics. In file in coreos/etcd, one can see the to pass to enable metrics. It would be great if the apiserver has a command line parameter, config/environment variable to enable etcd.clientv3 metrics.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is quite similar to
Yes, this is a good idea, in fact we just released a new version of the package required for this.
/assign
I can do it.
oh, maybe I should use the new features to implement, I will continue to modify it.
are you referring to injecting the registry to use? Yes I would very much appreciate that, even if we (for now) inject the , that will allow us to modify that more easily in the future.


Question: import ( "fmt" <ins> "strconv" </ins> "strings" "k8s.io/kubernetes/pkg/api" netsets "k8s.io/kubernetes/pkg/util/net/sets" <ins> "github.com/golang/glog" </ins> ) const (

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS m4.2xlarge OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-36-generic Install tools: kubeadm Others: 1 master and 1 minion What happened: This states that if using kube-proxy will only proxy requests to local endpoints, never forwarding traffic to other nodes and thereby preserving the original source IP address. However when I annotate my NodePort type service with this annotation I see the following error in the logs of kube-proxy of the node where the only pod of the service is running: After this message the service stops working outside of the cluster on the specified NodePort. What you expected to happen: I expected to see that only the with the pod on it serves traffic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
Sorry, I have found the issue, I had a setup where the none of the node names were the same as the hostname of the kube-proxy pod and this caused the issue, where none of the Pods running on a hosts were recognised as local pods. I have fixed this issue by manually setting the hostname with in kube-proxy. However I still insist that this error message is misleading and shouldn't be printed in case of NodePort type services, but only for just LoadBalancers.
I met the same problem in the aws cloud providers, by adding - the hostname-override to solve this problem. good


Question: name = "go_default_test", srcs = [ "attacher_test.go", <ins> "vsphere_volume_block_test.go", </ins> "vsphere_volume_test.go", ], embed = [":go_default_library"],

FEATURE REQUEST: vSphere volume plugin does not support volume device yet. Need to add volumeMode in PV spec as other plugin did. release 1.12
/sig storage


Question: goldClassName := "gold" now := time.Now() deletionTimestamp := metav1.Time{Time: time.Now().UTC().AddDate(-10, 0, 0)} <ins> snapshotAPIGroup := "snapshot.storage.k8s.io" </ins> testCases := []struct { name               string pvc                *corev1.PersistentVolumeClaim

Currently there's now way for a user to check what they entered in the DataSource field of a PVC create request. In the past, this wasn't as confusing because if one were to submit invalid info (unsupported Kind) the verification checks would fail and the request would be rejected with a helpful message indicating why. As of 1.14 that's no longer the case, a user is able to put anything they wish in these fields and the create is performed and they get a PVC, and there's no way for them to see what is or isn't set. Adding output of DataSource to would be useful, not only for invalid cases where the input might just get eaten, but also for users to have the ability to check for originating sources of their PVCs.
/sig storage
/assign
GitHub didn't allow me to assign the following users: me. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten


Question: "//vendor/k8s.io/apiserver/pkg/authentication/serviceaccount:go_default_library", "//vendor/k8s.io/apiserver/pkg/util/logs:go_default_library", "//vendor/k8s.io/client-go/kubernetes:go_default_library", <ins> "//vendor/k8s.io/client-go/plugin/pkg/client/auth:go_default_library", </ins> ], )

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: E2E keeps failing with internal Ginkgo failure of "Node 1 disappeared before completing BeforeSuite" on my PR. https://k8s- What you expected to happen: Not that. How to reproduce it (as minimally and precisely as possible): Run /retest on my PR. It's happened 5 times in a row. Anything else we need to know?: I am almost certain my change does not affect that test suite in any way.
cc
Worked on https://k8s- this morning: https://k8s-
Does this repo on an empty changelist?
Looks like it repos on ci: https://k8s-
Talked with gke dev oncall... internal dashboards look fine, doesn't seem like a gke issue.
Most interesting thing on is maybe the prow bump?
I'm seeing this too on my PR
Trying on a clean PR:
Did something bad happen between versions 342 and 346 on 1.11.0-alpha.2? CI fails to start starts: PR:
Maybe this has something to do with
I'm not sure. That was adding permissions back to nodes so they can delete themselves. It had broken downgrades, so it was reverted. It fixed this: https://k8s-
Testing the theory in
looks like no
https://k8s- so absolutely nothing changed on the test-infra side between the last pass at 9:10 and first failure at 9:55 (prow updated before 9a, same kubetest image, test-infra commit) Seems to be failing here : https://k8s- compared to passing run https://k8s- /unassign /assign
The client go auth plugins are by import. kubectl imports the gcp provider explicitly, but the e2e test apparently doesn't. is in the range when the e2e test started failing, and the description of that PR indicates it trimmed lots of dependencies. I'd look at whether the e2e test had only imported that auth plugin indirectly via one of those removed dependencies in that PR. If the e2e test requires that auth plugin, it should import it like kubectl and the kubeapiserver does:
Are you saying the e2e suite imports kubeadm code?

On Fri, May 11, 2018, 19:57 Jordan Liggitt <:
e2e imports kubeadm util: which imports preflight: which imported kubeapiserveroptions prior to : which imported: which imported the apiserver authenticator: which imported the auth plugins:
potential fix in
Thanks Great bug :^)


Question: runcmd: - systemctl daemon-reload - systemctl enable kube-master-installation.service <ins> - systemctl enable kube-master-internal-route.service </ins> - systemctl enable kube-master-configuration.service - systemctl enable kube-container-runtime-monitor.service - systemctl enable kubelet-monitor.service

If you create a private cluster via kube-up using provider GCE, you won't be able to restart your master nodes. If you restart you master, the nodes won't be able to connect to your master. This is due to this line - It gets executed on master on cluster-up, but it won't be re-executed on master restart. We should fix that, especially if we're to implement /sig scalability /priority important-soon
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /good-first-issue I'll ask whether someone in SIG Scalability public meeting would like to take on this one
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle stale /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello, Can I start work on this one as my first issue? :)
I don't see anyone else claiming it, so your contributions are welcome :)
let me know if you need any explanations here
Thank you :) I've made some changes and assign you can you tell me is this will be good approach?
Thanks, I replied in the PR


Question: }, } <del> kl.podManager.SetPods(pods) // Sync to create pod directories. kl.HandlePodSyncs(pods) for i := range pods { if !dirExists(kl.getPodDir(pods[i].UID)) { t.Errorf("expected directory to exist for pod %d", i) } } </del> <ins> syncAndVerifyPodDir(t, testKubelet, pods, pods, true) </ins> // Pod 1 failed, and pod 2 succeeded. None of the pod directories should be // deleted. kl.statusManager.SetPodStatus(pods[1], api.PodStatus{Phase: api.PodFailed}) kl.statusManager.SetPodStatus(pods[2], api.PodStatus{Phase: api.PodSucceeded}) <del> kl.HandlePodCleanups() for i := range pods { if !dirExists(kl.getPodDir(pods[i].UID)) { t.Errorf("expected directory to exist for pod %d", i) } </del> <ins> syncAndVerifyPodDir(t, testKubelet, pods, pods, true) } func TestDoesNotDeletePodDirsIfContainerIsRunning(t *testing.T) { testKubelet := newTestKubelet(t) testKubelet.fakeCadvisor.On("MachineInfo").Return(&cadvisorApi.MachineInfo{}, nil) testKubelet.fakeCadvisor.On("DockerImagesFsInfo").Return(cadvisorApiv2.FsInfo{}, nil) testKubelet.fakeCadvisor.On("RootFsInfo").Return(cadvisorApiv2.FsInfo{}, nil) runningPod := &kubecontainer.Pod{ ID:        "12345678", Name:      "pod1", Namespace: "ns", </ins> } <ins> apiPod := &api.Pod{ ObjectMeta: api.ObjectMeta{ UID:       runningPod.ID, Name:      runningPod.Name, Namespace: runningPod.Namespace, }, } // Sync once to create pod directory; confirm that the pod directory has // already been created. pods := []*api.Pod{apiPod} syncAndVerifyPodDir(t, testKubelet, pods, []*api.Pod{apiPod}, true) // Pretend the pod is deleted from apiserver, but is still active on the node. // The pod directory should not be removed. pods = []*api.Pod{} testKubelet.fakeRuntime.PodList = []*kubecontainer.Pod{runningPod} syncAndVerifyPodDir(t, testKubelet, pods, []*api.Pod{apiPod}, true) // The pod is deleted and also not active on the node. The pod directory // should be removed. pods = []*api.Pod{} testKubelet.fakeRuntime.PodList = []*kubecontainer.Pod{} syncAndVerifyPodDir(t, testKubelet, pods, []*api.Pod{apiPod}, false) </ins> }

Often observed in logs: However: Which probably means we'd be ok unmounting it: This probably also means we don't delete orphaned mirror pods, which is the next step in SyncPods ().


Question: // All nonsense below is temporary until we have go versions of these things. func runBash(stepName, bashFragment string) bool { <del> log.Printf("Running: %v", stepName) </del> cmd := exec.Command("bash", "-s") cmd.Stdin = strings.NewReader(bashWrap(bashFragment)) <ins> return finishRunning(stepName, cmd) } func run(stepName, cmdPath string) bool { return finishRunning(stepName, exec.Command(filepath.Join(*root, cmdPath))) } func finishRunning(stepName string, cmd *exec.Cmd) bool { log.Printf("Running: %v", stepName) </ins> stdout, stderr := bytes.NewBuffer(nil), bytes.NewBuffer(nil) if *verbose { cmd.Stdout = os.Stdout

building release, hopefully it's just binary missing; but i expect e2e-test to build binary automatically.
Steps to reproduce? /cc
Just run hack/e2e-, will fail. The _output/go/bin directory only has integration binary. (tried run first, still fails).
Did you try hack/build- first? e2e is in the default target list:
It's also in the client target list in the new build infrastructure:
Ah, I think that will do the trick. I didn't run hack/build-, as I saw e2e- output something like this (A little misleading if it won't build the binary). Re-running test now, will close the issue if it works. But it would be great if the binary will be built automatically.
I agree it should be built automatically. Otherwise, we'd at least have to update our e2e documentation.
Ah, yeah, I ran into this too and haven't fixed it yet. I this cmd/e2e to the build system, where does it end up? Apparently not in _output...


Question: // while the regular container did framework.ExpectNoError(results.HasRestarted(regular1)) }) <ins> ginkgo.When("a pod cannot terminate gracefully", func() { testPod := func(name string, gracePeriod int64) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "busybox", Image: imageutils.GetE2EImage(imageutils.BusyBox), Command: []string{ "sleep", "10000", }, }, }, TerminationGracePeriodSeconds: &gracePeriod, }, } } // To account for the time it takes to delete the pod, we add a 10 second // buffer. The 10 second buffer is arbitrary, but it should be enough to // account for the time it takes to delete the pod. bufferSeconds := int64(10) ginkgo.It("should respect termination grace period seconds [NodeConformance]", func() { client := e2epod.NewPodClient(f) gracePeriod := int64(30) ginkgo.By("creating a pod with a termination grace period seconds") pod := testPod("pod-termination-grace-period", gracePeriod) pod = client.Create(context.TODO(), pod) ginkgo.By("ensuring the pod is running") err := e2epod.WaitForPodRunningInNamespace(context.TODO(), f.ClientSet, pod) framework.ExpectNoError(err) ginkgo.By("deleting the pod gracefully") err = client.Delete(context.TODO(), pod.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err) ginkgo.By("ensuring the pod is terminated within the grace period seconds + buffer seconds") err = e2epod.WaitForPodNotFoundInNamespace(context.TODO(), f.ClientSet, pod.Name, pod.Namespace, time.Duration(gracePeriod+bufferSeconds)*time.Second) framework.ExpectNoError(err) }) ginkgo.It("should respect termination grace period seconds with long-running preStop hook [NodeConformance]", func() { client := e2epod.NewPodClient(f) gracePeriod := int64(30) ginkgo.By("creating a pod with a termination grace period seconds and long-running preStop hook") pod := testPod("pod-termination-grace-period", gracePeriod) pod.Spec.Containers[0].Lifecycle = &v1.Lifecycle{ PreStop: &v1.LifecycleHandler{ Exec: &v1.ExecAction{ Command: []string{ "sleep", "10000", }, }, }, } pod = client.Create(context.TODO(), pod) ginkgo.By("ensuring the pod is running") err := e2epod.WaitForPodRunningInNamespace(context.TODO(), f.ClientSet, pod) framework.ExpectNoError(err) ginkgo.By("deleting the pod gracefully") err = client.Delete(context.TODO(), pod.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err) ginkgo.By("ensuring the pod is terminated within the grace period seconds + buffer seconds") err = e2epod.WaitForPodNotFoundInNamespace(context.TODO(), f.ClientSet, pod.Name, pod.Namespace, time.Duration(gracePeriod+bufferSeconds)*time.Second) framework.ExpectNoError(err) }) }) </ins> })

Termination grace period includes the termination time of prestop hooks and containers in the pod. The total time that the pod terminates should always be < termination grace period seconds. A misconfigured pod with a preStopHook that hangs indefinitely and pod ignoring SIGTERM can result in the pod taking 2X (double) the termination grace period seconds to terminate. The time to terminate the pod should always be <= termination grace period seconds including prestop hook execution time as documented in 1) Apply the following pod 2) 3) Observe it actually took ~120s to terminate the pod not ~60s as would be expected. The kubelet logs for the pod The issue is that we are not taking into account the time taken by execution of the prestop hook when calculating the time remaining to kill the remaining containers in the pod. Reproduced on 1.26.0 <detailsn/a </details<details</details<details</details<details</details<details</details
/sig node
/sig node
The same error occurs in my local cluster, if you don't mind, I'd be happy to help fix the bug.
/assign
/triage accepted /priority important-longterm (I assume it is not a regression)


Question: "fmt" "time" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> utilerrors "k8s.io/apimachinery/pkg/util/errors" utilvalidation "k8s.io/apimachinery/pkg/util/validation" utilfeature "k8s.io/apiserver/pkg/util/feature"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --See for more context. What happened: When you set and , you'd expect container cfs quota period to be . However, unless you also set in the feature flags, the quota value will be silently ignored. What you expected to happen: At least a warning be logged if the feature is not enabled but is set. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /assign karan
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: apps "k8s.io/api/apps/v1beta1" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/client-go/tools/record" </ins> "k8s.io/kubernetes/pkg/controller/history" )

An issue to make bot happy for When StatefulSet finds any of its pods in phase Failed it will delete it and create it again. While doing so it will emit delete+create event which is really confusing for the user as he has no idea what's happening. /sig apps


Question: return nil } <del> var volumeLabels map[string]string if volume.Spec.AWSElasticBlockStore != nil { labels, err := l.findAWSEBSLabels(volume) if err != nil { return admission.NewForbidden(a, fmt.Errorf("error querying AWS EBS volume %s: %v", volume.Spec.AWSElasticBlockStore.VolumeID, err)) } volumeLabels = labels } if volume.Spec.GCEPersistentDisk != nil { labels, err := l.findGCEPDLabels(volume) if err != nil { return admission.NewForbidden(a, fmt.Errorf("error querying GCE PD volume %s: %v", volume.Spec.GCEPersistentDisk.PDName, err)) } volumeLabels = labels } if volume.Spec.AzureDisk != nil { labels, err := l.findAzureDiskLabels(volume) if err != nil { return admission.NewForbidden(a, fmt.Errorf("error querying AzureDisk volume %s: %v", volume.Spec.AzureDisk.DiskName, err)) } volumeLabels = labels } if volume.Spec.Cinder != nil { labels, err := l.findCinderDiskLabels(volume) if err != nil { return admission.NewForbidden(a, fmt.Errorf("error querying Cinder volume %s: %v", volume.Spec.Cinder.VolumeID, err)) } volumeLabels = labels } if volume.Spec.VsphereVolume != nil { labels, err := l.findVsphereVolumeLabels(volume) if err != nil { return admission.NewForbidden(a, fmt.Errorf("error querying vSphere Volume %s: %v", volume.Spec.VsphereVolume.VolumePath, err)) } volumeLabels = labels </del> <ins> volumeLabels, err := l.findVolumeLabels(volume) if err != nil { return admission.NewForbidden(a, err) </ins> } requirements := make([]api.NodeSelectorRequirement, 0) if len(volumeLabels) != 0 { if volume.Labels == nil {

API server latency for PV creation depends on underlying cloud API latency. We have admission plugin that examines incoming PV and adds topology labels and node selector: When the cluster (and cloud) are under pressure, e.g. lot of PVCs are created and dynamically provisioned at one time and attached to nodes, API server latency increases. We've seen cases when we ran out of cloud API request quota and the admission plugin needed to sleep for couple of seconds (on AWS). There should be an easy fix: check, if the created PV already has all necessary labels and topology information and ask the cloud for them only when they're missing. Dynamically provisioned volumes already have these labels. The admission plugin will then ask the cloud only for manually provisioned PVs, which we believe is just a fraction of all PVs in the cluster. There is one glitch: previously, when user created a PV with wrong topology labels, the admission plugin corrected them. With the fix suggested above, this PV won't be corrected. /sig storage cc
Implementation of the above:


Question: // restart may think these old pods are consuming CPU and we // will get an OutOfCpu error. ginkgo.By("verifying restartNever pods succeed and restartAlways pods stay running") <del> postRestartRunningPods := waitForPods(f, numAllPods, startTimeout) </del> <ins> postRestartRunningPods := waitForPods(f, numAllPods, recoverTimeout) </ins> if len(postRestartRunningPods) < numAllPods { framework.Failf("less pods are running after node restart, got %d but expected %d", len(postRestartRunningPods), numAllPods) }

node-kubelet-serial Kubelet should correctly account for terminated pods after restart Since test was created, flakes 10-20% of the time. Not entirely clear why, I haven't looked. No response /assign /sig node
Introduced by , maybe we should use to replace See the log below, a pod becomes Succeed may exhaust tens of seconds
/triage accepted
Yeah I only tested this locally and not in CI, it seems that in CI the pods take much longer. Timings just need to be tweaked.
/reopen still a number of flakes since the attached PR merged, I don't think that fixed it.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from this link, I see all suites passed, which flakes do you mean?
I see this failure in
so this problem should have been fixed by
From gubernator: In all cases, it's timing out. I'll try upping the startTimeout.
/reopen still flaking
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Luckily the kubelet log is clear as to what's going on. From (flake ) The test should work fine if we drop the CPU request a bit, let me tweak it to make up for the possible extra 100m scheduled. I'm not sure what's causing that (aren't these tests serial? what the heck is getting scheduled) but we should be resilient to it. Also, needs to be fixed. This test will no longer detect the regression it was designed to show after that change. I will revert it and also fix the resource requirements and that should fix this test.
/milestone v1.23 need to fix this before test freeze