Question: readonly green=$(tput setaf 2) kube::test::clear_all() { <del> kubectl delete rc,pods --all </del> <ins> kubectl delete "${kube_flags[@]}" rc,pods --all </ins> } kube::test::get_object_assert() {

This works: Are we using /var/run for testing?
Curiously travis shows a similar error message regarding /var/run/kubernetes, but it isn't failing: I suspect the /var/run/kubernetes issue is not what's causing the failure here.
However, I am able to reproduce the failure on my machine. I note that in your case, the fatal error says For me, it says The ip addresses showing through here (which appear to be in the GCE space?) are worrying.
It seems it still failing. I've run it locally and it failed with the following error: The aforementioned line: I think it is expected that there are some pods started by k8s (e.g. elasticsearch). Does it mean the test has a bug?
Strangely the same code () passes tests on travis (https://travis-)
I think you're reading it backwards. The test is expecting no pods to be started, but finding the k8s add-on pods, which is normal if it were running against a normal master. In Travis, it's a synthetic environment with no add-ons. This should be assigned to whoever the checks (on mobile.. I know I merged the PR but don't remember who :)). We need to exclude the add-ons (perhaps by finally getting around to putting them in a separate namespace).
the assertions, so CC'ing for comment.
I looked into this with and it seems that test- may fail locally depending on the kubeconfig. Some commands are using defaults for kubectl and thus sometimes they contact another cluster. I'll send a PR to fix this.


Question: if p, err := q.Pop(); err != nil || p != &highPriNominatedPod { t.Errorf("Expected: %v after Pop, but got: %v", highPriNominatedPod.Name, p.Name) } <del> if p, err := q.Pop(); err != nil || p != &medPriorityPod { t.Errorf("Expected: %v after Pop, but got: %v", medPriorityPod.Name, p.Name) } </del> if len(q.nominatedPods.nominatedPods) != 1 { t.Errorf("Expected nomindatePods to have one element: %v", q.nominatedPods) }

We've noticed a strange behavior of the scheduler in GCE 5K node scalability tests. There were periods of time, lasting about 6s, when scheduler didn't schedule any new pods. It's ruining our 5s pod-startup-latency SLO. See the example below, pod density-latency-pod-2325-dgqrv created on 09:08:59, scheduler didn't schedule it until 09:09:06. An interesting pattern can be found in the around that time. Before 09:09:00 scheduler schedules ~5 pods / s. Then it goes silent for about 6s and at 09:09:06 it outputs about 30 "bound successfully" statements and continues scheduling pods with ~5 pods / s pace. We checked api-server logs for that silent period of time and it turned out that scheduler was doing something, it was constantly sending GET and PUT request for a metrics-server and heapster pod it was trying to schedule. ! We dug deeper and found that scheduler was actually trying to schedule these two pods for over 12h, below is the plot of number of calls per minute that scheduler issued to api-server for these two pods: ! As we can see there are long periods of time when scheduler issues over 1500 requests per minute. That seems strangely a lot given that the requests are only for the two pods. During the whole run scheduler issued over 400K GETs and over 200K PUTs for these two pods. ! We figured out that scheduler wasn't able to schedule these two pods because there was not enough resources in the cluster. While it's understandable that scheduler wasn't able to schedule the pods, I'd expect scheduler to not spam the system with so many retries, which, as we can see in this example, can potentially block / slow-down scheduling of other pods. I've increased the scheduler log verbosity level in , we should have more logs from the tomorrow's run. Will post them here once available.
/sig scheduling
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig scheduling </details
/assign for triage
What is the version of your cluster? If you are using the HEAD, we have a back-off mechanism to the scheduler to delay rescheduling unschedulable pods.

It starts with 1 second delay and doubles it every time the pod is not schedulable up to 10 seconds.
It's from head, so apparently something doesn't work...
Yes, we're using head. The test run on Jan 21 and the cluster was built from head on the same day (commit to be more precise). As you can see in the logs it doesn't look like scheduler is using any back-off mechanism. Definitely not the one you described, it issues requests to api server much more often than 1 per second per pod. !
An update on more logs - we won't have more logs. We tried increasing scheduler's log level to v4 and it resulted in the log growing by few orders of magnitude and it was clogged with mostly useless stuff, e.g. it logged "Node gce-scale-cluster-minion-group-XXX is a potential node for preemption." over 250 000 000 times. We reverted the log level change in If more scheduler logs are needed to debug this we probably should add some new, more useful, ones.
cc/
/assign I can look into this next Tuesday
Friendly ping, any updates?
Whoops. Thanks for the ping. Log level v4 should not be needed. Although the log line you called out should probably get pushed into V9... First off, that's some really great debugging. Thanks for the detailed report. If the scheduler is getting into tight busy loops, that could in theory starve the scheduling thread, causing the observed 6 second pause in scheduling. The bug is in  A pod only gets marked for backoff if . And can increment very quickly in a high churn cluster. This would result in a pod being retried in a tight loop. One (unrelated) observation that stood out to me was this: This is being printed by the package and shows a trace with no steps logged to it, which means that we are hitting one of these four error cases in . Most likely, the PVC for those heapster & metrics-server pods are in a bad/unexpected state.
I think fixes the root problem here. If we eliminate the API server spam, the main scheduling thread hopefully will not get blocked for 6 second periods. Are you able to test with this patch?
Thanks Jonathan!

I've reserved a scale-testing slot for Thursday (Feb 21) to test the fix. Will update this issue once I have the results.
I reproduced the issue (non-schedulable heapster pod due to not enough resources) on a 5K node cluster. Then I run the load test, first on the version without the fix and the on the version with the fix. Below are the graphs of the number of scheduler requests (for the unschedulable pod) per minute. Without fix: ! With fix: ! As we can see on the graphs, the fix definitely helps, without it there were minutes were scheduler was calling api server up to 2K times per minute (2K QPM). With the fix it didn't exceed 100 QPM. But even with the fix the average QPM is ~60 (per single pod) which is 1 query per second per pod on average. is average 60 QPM (1 QPS) reasonable here or should we expect something less from the rate-limiting mechanism?
By default the scheduler rate limit is 20 qps for all scheduler operations. Default backoff delay for unschedulable pods is 1 second and goes exponentially higher every time a pod is retried, but does not exceed 10 seconds. This 10 second max backoff time also means that if a pod is not retried for 10 seconds, for example because there are many other pods in front of it in the queue, its backoff time is reset to 1 second. So the next time it is retried and is found unschedulable it will be backed off for 1 second. I am basically trying to say that 1 qps is not unreasonable. We also an optimization () that should lower qps for unschedulable pods further down. This last PR got merged only a few hours ago. With that PR merged, I would expect to see even lower qps as the scheduler will not update pod status every time it retries scheduling a pod.
Thanks Bobby! That makes sense. In that case I believe we can close this issue once merges.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks for fixing!


Question: fi fi done <del> if [[ "${KUBE_SKIP_UPDATE}" == "y" ]]; then return fi # update and install components as needed if [[ "${KUBE_PROMPT_FOR_UPDATE}" != "y" ]]; then gcloud_prompt="-q" fi local sudo_prefix="" if [ ! -w $(dirname `which gcloud`) ]; then sudo_prefix="sudo" fi ${sudo_prefix} gcloud ${gcloud_prompt:-} components install alpha || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components install beta || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components update || true </del> <ins> update-or-verify-gcloud </ins> } # Create a temp dir that'll be deleted at the end of this bash session.

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.


Question: r.runtimeClientV1alpha2 = runtimeapiV1alpha2.NewRuntimeServiceClient(conn) } else { <del> return fmt.Errorf("unable to determine runtime API version: %w", err) </del> <ins> return fmt.Errorf("unable to determine runtime API version with %q, or API is not implemented: %w", endpoint, err) </ins> } return nil

When installing the latest component, got the error: Then the flag , got error: It is confusing to see these error msgs at the first glance, need to make it more clearly. Add the required field validation, and add more supportive error message. Download the latest v1.25.0 binary, and install it with systemd service or just manually start the binary, will reproduce these errors. <details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
kubelet 1.24+ is not working with the docker, unless using Are you using it? If not, try switching to other available container runtimes. Please re-open if you have further questions /close
Closing this issue. <detailsIn response to : kubelet 1.24+ is not working with the docker, unless using Are you using it? If not, try switching to other available container runtimes. Please re-open if you have further questions Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, I got to know the , so need to add runtime validation by the fix PR: PTAL thanks.


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "io/ioutil" "os" "path" "testing" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/sets" ) type testCases struct { name     string err      bool expected sets.String volname  string pod      v1.Pod } func TestGetNestedMountpoints(t *testing.T) { var ( testNamespace = "test_namespace" testPodUID    = types.UID("test_pod_uid") ) tc := []testCases{ { name:     "Simple Pod", err:      false, expected: sets.NewString(), volname:  "vol1", pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: testNamespace, UID:       testPodUID, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { VolumeMounts: []v1.VolumeMount{ {MountPath: "/dir", Name: "vol1"}, }, }, }, }, }, }, { name:     "Simple Nested Pod", err:      false, expected: sets.NewString("nested"), volname:  "vol1", pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: testNamespace, UID:       testPodUID, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { VolumeMounts: []v1.VolumeMount{ {MountPath: "/dir", Name: "vol1"}, {MountPath: "/dir/nested", Name: "vol2"}, }, }, }, }, }, }, { name:     "Unsorted Nested Pod", err:      false, expected: sets.NewString("nested", "nested2"), volname:  "vol1", pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: testNamespace, UID:       testPodUID, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { VolumeMounts: []v1.VolumeMount{ {MountPath: "/dir/nested/double", Name: "vol3"}, {MountPath: "/ignore", Name: "vol4"}, {MountPath: "/dir/nested", Name: "vol2"}, {MountPath: "/ignore2", Name: "vol5"}, {MountPath: "/dir", Name: "vol1"}, {MountPath: "/dir/nested2", Name: "vol3"}, }, }, }, }, }, }, { name:     "Multiple vol1 mounts Pod", err:      false, expected: sets.NewString("nested", "nested2"), volname:  "vol1", pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: testNamespace, UID:       testPodUID, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { VolumeMounts: []v1.VolumeMount{ {MountPath: "/dir", Name: "vol1"}, {MountPath: "/dir/nested", Name: "vol2"}, {MountPath: "/ignore", Name: "vol4"}, {MountPath: "/other", Name: "vol1"}, {MountPath: "/other/nested2", Name: "vol3"}, }, }, }, }, }, }, { name:     "Big Pod", err:      false, volname:  "vol1", expected: sets.NewString("sub1/sub2/sub3", "sub1/sub2/sub4", "sub1/sub2/sub6", "sub"), pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: testNamespace, UID:       testPodUID, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { VolumeMounts: []v1.VolumeMount{ {MountPath: "/mnt", Name: "vol1"}, {MountPath: "/ignore", Name: "vol2"}, {MountPath: "/mnt/sub1/sub2/sub3", Name: "vol3"}, {MountPath: "/mnt/sub1/sub2/sub4", Name: "vol4"}, {MountPath: "/mnt/sub1/sub2/sub4/skip", Name: "vol5"}, {MountPath: "/mnt/sub1/sub2/sub4/skip2", Name: "vol5a"}, {MountPath: "/mnt/sub1/sub2/sub6", Name: "vol6"}, {MountPath: "/mnt7", Name: "vol7"}, }, }, }, InitContainers: []v1.Container{ { VolumeMounts: []v1.VolumeMount{ {MountPath: "/mnt/dir", Name: "vol1"}, {MountPath: "/mnt/dir_ignore", Name: "vol8"}, {MountPath: "/ignore", Name: "vol9"}, {MountPath: "/mnt/dir/sub", Name: "vol11"}, }, }, }, }, }, }, { name:     "Naughty Pod", err:      true, expected: nil, volname:  "vol1", pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: testNamespace, UID:       testPodUID, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { VolumeMounts: []v1.VolumeMount{ {MountPath: "foo/../../dir", Name: "vol1"}, {MountPath: "foo/../../dir/skip", Name: "vol10"}, }, }, }, }, }, }, } for _, test := range tc { dir, err := ioutil.TempDir("", "TestMakeNestedMountpoints.") if err != nil { t.Errorf("Unexpected error trying to create temp directory: %v", err) return } defer os.RemoveAll(dir) rootdir := path.Join(dir, "vol") err = os.Mkdir(rootdir, 0755) if err != nil { t.Errorf("Unexpected error trying to create temp root directory: %v", err) return } dirs, err := getNestedMountpoints(test.volname, rootdir, test.pod) if test.err { if err == nil { t.Errorf("%v: expected error, got nil", test.name) } continue } else { if err != nil { t.Errorf("%v: expected no error, got %v", test.name, err) continue } } actual := sets.NewString(dirs...) if !test.expected.Equal(actual) { t.Errorf("%v: unexpected nested directories created:nexpected: %vn     got: %v", test.name, test.expected, actual) } } } </ins>

This bug affects any pod that has a container with a nested volume (any type) beneath an API data volume, such as a secret, configMap, downwardAPI or projected volume. In the description and reproduction details below I use a secret volume, but any of the four will demonstrate this behavior. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a pod with nested secrets volumes (e.g. one volume mounted at and one mounted at ), the container fails to start with an error like: What you expected to happen: The container should start with both and mounted. How to reproduce it (as minimally and precisely as possible): out master, any version since a file with 2 secrets and a pod that makes volumes for the 2 secrets, with one of them mounted within the other: the 2 secrets and the pod. Assuming you named the above file : Anything else we need to know?: This may work for other runtimes if they create mountpoints outside of the sandbox. But it's definitely broken with Docker as the runtime.
This should be fixed by /sig storage /sig node /assign


Question: # Deduplicate to ensure the intersection trick with kube::golang::dups # is not defeated by duplicates in user input. <del> mapfile -t platforms <<< "$(kube::golang::dedup "${platforms[@]}")" </del> <ins> kube::util::read-array platforms < <(kube::golang::dedup "${platforms[@]}") </ins> # Use kube::golang::dups to restrict the builds to the platforms in # KUBE_SUPPORTED_*_PLATFORMS. Items should only appear at most once in each # set, so if they appear twice after the merge they are in the intersection. <del> mapfile -t KUBE_SERVER_PLATFORMS <<< "$(kube::golang::dups  </del> <ins> kube::util::read-array KUBE_SERVER_PLATFORMS < <(kube::golang::dups  </ins> "${platforms[@]}"  "${KUBE_SUPPORTED_SERVER_PLATFORMS[@]}"  <del> )" </del> <ins> ) </ins> readonly KUBE_SERVER_PLATFORMS <del> mapfile -t KUBE_NODE_PLATFORMS <<< "$(kube::golang::dups  </del> <ins> kube::util::read-array KUBE_NODE_PLATFORMS < <(kube::golang::dups  </ins> "${platforms[@]}"  "${KUBE_SUPPORTED_NODE_PLATFORMS[@]}"  <del> )" </del> <ins> ) </ins> readonly KUBE_NODE_PLATFORMS <del> mapfile -t KUBE_TEST_PLATFORMS <<< "$(kube::golang::dups  </del> <ins> kube::util::read-array KUBE_TEST_PLATFORMS < <(kube::golang::dups  </ins> "${platforms[@]}"  "${KUBE_SUPPORTED_TEST_PLATFORMS[@]}"  <del> )" </del> <ins> ) </ins> readonly KUBE_TEST_PLATFORMS <del> mapfile -t KUBE_CLIENT_PLATFORMS <<< "$(kube::golang::dups  </del> <ins> kube::util::read-array KUBE_CLIENT_PLATFORMS < <(kube::golang::dups  </ins> "${platforms[@]}"  "${KUBE_SUPPORTED_CLIENT_PLATFORMS[@]}"  <del> )" </del> <ins> ) </ins> readonly KUBE_CLIENT_PLATFORMS elif [[ "${KUBE_FASTBUILD:-}" == "true" ]]; then

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --This issue was moved from What happened: building kubernetes with make fails on mac after What you expected to happen: building should not fail How to reproduce it (as minimally and precisely as possible): build from a mac Anything else we need to know?: mapfile is not available on macOS bash. we should have a version check at minimum if we're going to use unsupported features (some of the code already does this) cc
/sig release
Ugh. Does the containerized build at least still work on Mac? Ideally we'd stick to that as the recommended approach, since we can at least guarantee a correct environment. I guess this is really Apple's fault, for refusing to upgrade the bash version on Macs (so they can avoid GPL) - the default on most Macs is 3.2.57, which dates to 2007 according to this blog post: I don't personally think we should support something that old. None of our CI uses a bash that old. It's ripe for breakage. You can install a newer version of bash via homebrew.
the containerized build does not because this code runs on the host. we should put a version check then, so we get a more understandable error.
Up until this point we have supported it by using instead, macOS's old system bash is annoying, but errors from deep within our bash waste more time supporting confused developers
IIRC even in CI you ran into a bash version problem in fact just yesterday (?) we could add a snippet like this to the main build if we don't want to switch to :
I worry about requiring brew and can imagine some folks systems might be managed in a way that disallow this. Still it's reaaaaally painful having to debug and trying to keep things running with the crufty OSX dev userspace. Given there are multiple snippets of brew install suggestions, it would be better imho to put them in one common top level place (ie: )
fair, On systems with restricted at least should work, if that doesn't work i'd expect the setup is just too restrictive to build at all (?)

The check doesn't have to require homebrew per se either, even just "Sorry building on macOS is unsupported" is more helpful than:
Fix inbound:
/assign


Question: gvr("flowcontrol.apiserver.k8s.io", "v1alpha1", "prioritylevelconfigurations"): { Stub:             `{"metadata": {"name": "conf1"}, "spec": {"type": "Limited", "limited": {"assuredConcurrencyShares":3, "limitResponse": {"type": "Reject"}}}}`, ExpectedEtcdPath: "/registry/prioritylevelconfigurations/conf1", <del> ExpectedGVK:      gvkP("flowcontrol.apiserver.k8s.io", "v1beta2", "PriorityLevelConfiguration"), </del> <ins> ExpectedGVK:      gvkP("flowcontrol.apiserver.k8s.io", "v1beta3", "PriorityLevelConfiguration"), </ins> }, // --

adds new version for in 1.26. For 1.26, we are setting the storage version to because vN-1 understands that level. In 1.27, we should change the storage version from to .
/sig api-machinery
/cc
/cc
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: } } } <ins> func TestValidatePriorityConfigOverFlow(t *testing.T) { tests := []struct { description string configs     []algorithm.PriorityConfig expected    bool }{ { description: "one of the weights is MaxInt", configs:     []algorithm.PriorityConfig{{Weight: api.MaxInt}, {Weight: 5}}, expected:    true, }, { description: "after multiplication with MaxPriority the weight is larger than MaxWeight", configs:     []algorithm.PriorityConfig{{Weight: api.MaxInt/api.MaxPriority + api.MaxPriority}, {Weight: 5}}, expected:    true, }, { description: "normal weights", configs:     []algorithm.PriorityConfig{{Weight: 10000}, {Weight: 5}}, expected:    false, }, } for _, test := range tests { err := validateSelectedConfigs(test.configs) if test.expected { if err == nil { t.Errorf("Expected Overflow for %s", test.description) } } else { if err != nil { t.Errorf("Did not expect an overflow for %s", test.description) } } } } </ins>

Weight is an int (see ). Priority can take the range 1-10. Total priority for a node is sum of weight  priority for each priority function, and the sum is accumulated into an int (see ). So obviously the total priority can overflow if someone sets the weight(s) too high. The weights of all priority functions are compiled in (or specified in config file), so we can check to make sure maximum total priority can't overflow an int (take sum over all priority functions of 10  weight of that priority function, and make sure that's smaller than int(MaxUint 1) which according to StackOverflow is the way to find out what is max int (math package has max Int32, Int64, etc. but not max Int). Note that scheduler extender is OK because although the policy runs in a separate process, the weight is specified in scheduler config file (ExtenderConfig struct) and the priority is bounded 1-10 as usual.
We really should also make "score" be an abstract datatype rather than a raw int that is manipulated directly, so we can do things like make sure it is in the range 1-10 (or 0-10 if that's what we want); today the range is inconistent (we say 1-10, but sometimes use 0; probably we should change it to 0-9..).
I volunteer myself to do this.
I think we can check weight overflow in after we get all PriorityConfig. what do you think about this?
As for "score" I think we can use
Fixed by - closing.
Tim that PR was closed, not merged. I don't think this is fixed yet.
whoops, good catch, just removing milestone then. There was a shift that happened yesterday where a bunch of items suddenly came into the 1.6 query.
do we still want this fixed? I can re-pick this.
AFAIK it's still broken. cc/
Ok, I will open a PR tomorrow.


Question: } } <del> informerFactory := informers.NewSharedInformerFactory(s.Client, s.ConfigSyncPeriod) </del> <ins> informerFactory := informers.NewSharedInformerFactoryWithOptions(s.Client, s.ConfigSyncPeriod, informers.WithTweakListOptions(func(options *v1meta.ListOptions) { options.LabelSelector = "!service.kubernetes.io/service-proxy-name" })) </ins> // Create configs (i.e. Watches for Services and Endpoints) // Note: RegisterHandler() calls need to happen before creation of Sources because sources

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Implement the feature described here: Why is this needed: In a cluster that has a service mesh a lot of the work being done by kube-proxy is redundant and wasted. Specifically, services that are only reached via other services in the mesh will never use the service abstraction implemented by kube-proxy in iptables (or ipvs). By informing the kube-proxy of this, we can lighten the work it is doing and the burden on its proxy backend. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/sig network


Question: }, Status: networkingv1.IngressStatus{LoadBalancer: v1.LoadBalancerStatus{}}, } <ins> ingress1 := ingTemplate.DeepCopy() ingress1.Spec.Rules[0].Host = "host1.bar.com" ingress2 := ingTemplate.DeepCopy() ingress2.Spec.Rules[0].Host = "host2.bar.com" ingress3 := ingTemplate.DeepCopy() ingress3.Spec.Rules[0].Host = "host3.bar.com" </ins> // Discovery ginkgo.By("getting /apis") {

We run sonobuoy tests on production running clusters whenever we upgrade our clusters. We observe that fails when we already have nginx-ingress controller running in the cluster. The newer versions of nginx-ingress controller comes up with a validating admission webhook that validates the ingress definitions. In this specific test, we are creating 3 ingress objects with the same path and the webhook do not permit to create them (the first one is created but the other two would fail) and then the job fails. The tests shouldn't fail and it requires we create ingress objects using different paths instead. Install nginx-ingress controller from Run e2e test. Test fails. No response =1.19 n/a n/a n/a n/a n/a
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing /area e2e-test-framework
This is a duplicate, see the comment author of the ingress nginx controller explaining why this is working as expected Edit check again, maybe is not a duplicate
Thanks for pointing this. It's infact the same issue. But the fix above should hopefully help since it would create those ingress objects with different paths and the validating webhook for ingress controller would permit creating objects in that case.
yeah sorry, just checked quickly from the mobile, I will take a look Thanks


Question: import ( "errors" "fmt" <ins> "io/ioutil" "path" </ins> "testing" "time" "github.com/google/go-cmp/cmp" <ins> "github.com/spf13/pflag" </ins> "github.com/stretchr/testify/assert" <ins> "github.com/stretchr/testify/require" </ins> v1 "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/api/resource" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" clientsetfake "k8s.io/client-go/kubernetes/fake" componentbaseconfig "k8s.io/component-base/config"

introduced configuration of additional logging aspects (e.g. format) through command line flags and config file. This was merged for Kubernetes 1.28. Now the following invocation doesn't work as expected when the config file doesn't configure logging: It should run with verbosity 4. Instead it runs with verbosity 0, from the config. Run with this config file: No response 28 rc n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The problem is caused by this line: At that point, is 4 (from the command line flag), and then it gets overwritten with == 0. The normal approach would be: set up config with defaults modify config through flags modify config further by parsing a config file into it. Can kube-proxy do this? It has the downside that a config file can only overwrite fields if they are in the file. A config field doesn't get reset to defaults just by parsing the file. /cc
it looks like kube-proxy special-cases the to let a flag override config... can/should it do the same thing for verbosity? are there any other flags like this?
I don't think that's the normal approach... it's more normal to make flags and config file mutually exclusive. That's difficult for things which started as flags and moved into config
/sig network /sig instrumentation /kind bug /kind regression /milestone v1.28
let's go with and warn it and comment as much as possible, I was going with something like but Patrick is working on it so I assign to him the issue
/assign
Hey folks, v1.28 Bug Triage Lead here Just wanted to know if this is already tracked by SIG leads and someone already working actively fixing it and set a priority?
Perhaps I should have said "this is how component-base/logs was meant to be used". Somehow it worked for kubelet.

Here's my preliminary solution: It needs more testing, but I wanted to share it early because I am running out of time today.
it is , I can take over Patrick's work in case is needed
It would be nice if the logs API could distinguish between and , how about make to a pointer? But this change is supposed to be a breaking change, I'm not sure of the scope of the impact. For side, if set and at the same time, should the command flag override the config file or warn it and be discarded? I think it should be overridden.
xref:


Question: fi env-to-grains "runtime_config" <ins> env-to-grains "kube_user" </ins> } function salt-node-role() {

abac-authz- hardcodes "admin" instead of using the $KUBE_USER variable. This means that "admin" is the only basic-auth username that will work with that abac config.
To workaround until it is fixed, you can add a line to the master's abac-authz- file: and then restart the kube-apiserver (docker kill it, and let it restart).


Question: # CNI_BIN_DIR is the path to network plugin config files. CNI_BIN_DIR=${CNI_BIN_DIR:-""} <del> # KUBELET_KUBECONFIG_DIR is the path to a dir for the kubelet's kubeconfig file </del> <ins> # KUBELET_KUBECONFIG is the path to a kubeconfig file, specifying how to connect to the API server. </ins> KUBELET_KUBECONFIG=${KUBELET_KUBECONFIG:-"/var/lib/kubelet/kubeconfig"} # Creates a kubeconfig file for the kubelet.

/kind bug What happened: From doest not find kubeconfig file What you expected to happen: To find kubeconfig file How to reproduce it (as minimally and precisely as possible): from conformance file Anything else we need to know?: Line 185 : Starting kubelet with args But does not exist. Using from line 80 seems to work properly Environment: Kubernetes version (use ):Cloud provider or hardware configuration: hardware configuration OS (e.g. from /etc/os-release): Fedora 28 Server Kernel (e.g. ): Linux 4.17.12-200.fc28.x86_64
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area conformance


Question: package app import ( <ins> "net/http" </ins> "testing" "time"

In we observed a crashlooping controller cause a cluster cold boot failure, as controller-manager wasn't able to stay up long enough to run the deployment that would have fixed the problem. We should audit everything that runs in kube-controller-manager and ensure that they all fail gracefully enough to prevent crashloops, or make them individually restart in that case, instead of the process. Longer term, we should consider classifying the controllers by run level or bootstrap order (), and splitting them into separate binaries or changing the start-up order in kcm to make sure this sort of thing doesn't happen in the future.
Same issue with metrics-server, using minikube. Also reproduced on our test cluster. Steps to reproduce: Set up localkube Install metrics-server Stop minikube: sudo minikube stop Start minikube: sudo -E minikube start --vm-driver=none Expected: Kubernetes is running Actual: Kubernetes is dead Logs: Workaround: To fix the system, we must delete the metrics-server: Now kubernetes will start normally. After that we can reinstall the metrics server, and it will be ok.
Related:
This will be a hard problem to fix permanently. In all likelihood, we should stop allowing Fatalf() in kubernetes/kubernetes, but as you can see we do this to ourselves right here...
/kind bug
I also think this bug would benefit from an e2e test where we misconfigure things in a variety of ways and verify that KCM still comes up and stays up: [ ] misconfigured apiservice [ ] broken service attached to that apiservice [ ] RBAC rules that disallow things. maybe everything! [ ] webhooks that disallow everything [ ] yr example here
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: }, } <del> resourceEncoding := serverstorage.NewDefaultResourceEncodingConfig(legacyscheme.Scheme) // This configures the testing apiserver the same way the real apiserver is // configured. The storage versions of these resources are different // from the storage versions of other resources in their group. resourceEncodingOverrides := []schema.GroupVersionResource{ batch.Resource("cronjobs").WithVersion("v1beta1"), apisstorage.Resource("volumeattachments").WithVersion("v1beta1"), networking.Resource("ingresses").WithVersion("v1beta1"), } resourceEncoding = resourceconfig.MergeResourceEncodingConfigs(resourceEncoding, resourceEncodingOverrides) storageFactory := serverstorage.NewDefaultStorageFactory(*storageConfig, "application/vnd.kubernetes.protobuf", legacyscheme.Codecs, resourceEncoding, DefaultAPIResourceConfigSource(), nil) </del> <ins> storageFactoryConfig := kubeapiserver.NewStorageFactoryConfig() resourceEncoding := resourceconfig.MergeResourceEncodingConfigs(storageFactoryConfig.DefaultResourceEncoding, storageFactoryConfig.ResourceEncodingOverrides) storageFactory := serverstorage.NewDefaultStorageFactory(*storageConfig, "application/vnd.kubernetes.protobuf", storageFactoryConfig.Serializer, resourceEncoding, DefaultAPIResourceConfigSource(), nil) </ins> etcdOptions := options.NewEtcdOptions(storageConfig) // unit tests don't need watch cache and it leaks lots of goroutines with etcd testing functions during unit tests

Storage version defaults to the preferred version for a given group, but can be overridden when constructing API server storage. This is done here: Reasons the preferred version might be overridden: A given resource/kind does not yet exist in the preferred version (like when introducing a new alpha type in an existing API group with more stable versions) A given resource/kind is continuing to store in etcd in an older version for compatibility with n-1 API servers The etcd integration test verifies that we're actually storing the correct versions in etcd, but the computation of the StorageVersionHash is not honoring the overridden version. on master, this code reports v1 CSIStorageCapacity: even though the etcd test verifies we're still persisting v1beta1 /kind bug /priority important-soon /sig api-machinery cc Originally posted by in
/cc
cc
hi I have found that on points to but here the points to here is screenshot from debug console ! As I am still new to the codebase, I am trying to understand why it is happening so. But we can say that because of this reason, test case below does not failed because the expected hash we put in file matches correctly with the onc we create in at line 404 I also believe that the value corresponding to key has been wrongly put same as Correct values are as follows. One caveat here is, hashes are calculated with appending resource but the key for lookup is based on resource . I hope this information will help in locating the issue. Meanwhile, I will also try to look why in is pointing to even though we override.
/assign
The problem reported by this issue should not exist, on apiserver startup, we can get the right overrided version v1beta1 for the csistoragecapacities from , since the code below overrided the storage version ! But in the test , it did not override the storage version, see the code below so this test case reports v1 CSIStorageCapacity is right. do you think that we should change the testcase to override the storage version again to keep consistent and dispel the confusion for StorageVersionHash ?
Yes you are right. If we add below testcase will surely fail as the hash will not match in this case.

So, we need to update hash in to
ah, good catch... so this is a test-only issue, but still one we need to resolve, so the unit test reflects reality of storage versions when you actually start a kube-apiserver
As this pr would remove the storageversion override so after this pr merged, the testcase would be just right, then we don't need another pr to fix the testcase again, right?
But alteast we should update the correct hash value corresponds to in
the test would still be exercising preferred version ordering without any overrides, which would still be incorrect, and would resurface problems as soon as another override is
/triage accepted


Question: namespace: kube-system labels: k8s-app: fluentd-gcp-scaler <del> version: v0.2.0 </del> <ins> version: v0.3.0 </ins> addonmanager.kubernetes.io/mode: Reconcile spec: selector:

This test is failing in gce serial suite: http://k8s- /sig cluster-lifecycle /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign xref
/milestone v1.10
this also fails in upgrade suite xref
looks like is now green. guess it's still flaky
not from http://k8s-
oh, probably pasted wrong link in the issue body? my bad
ah cool.
Mar 7 16:52:36.943: At least one pod wasn't running and ready or succeeded at test start. ^ The preconditions are not met at the start of the test... and the pods state shows as pending. Always seems to be a fluentd pod on the master node doesn't run. Are things tainted properly? /cc & as they are the test authors.
/assign /assign
GitHub didn't allow me to assign the following users: mbforbes. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
FWIW, I didn't author the test, and the test didn't even run since it failed the precondition.... /unassign /assign /assign fluentd pods are pending on the master nodes.
GitHub didn't allow me to assign the following users: bmoyles0117. Note that only and repo collaborators can be assigned. <detailsIn response to : FWIW, I didn't author the test, and the test didn't even run since it failed the precondition.... /assign /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
status?
ping
Looking into it
I have some problems with running e2e tests locally Could you please take a look? I will return to this tomorrow Europe time
I'm not sure what exactly is failing here. I was hoping to see if the two PRs we merged previously would have a positive impact on these failures. Is there any way of seeing whether or not those two PRs did or did not have a positive impact?

The two PRs for context are:
made DaemonSet controller schedules pods on "unschedulable" nodes. The changes lets the fluentd pod to be scheduled on the master, but there is no capacity on the master node to run this pod. /cc
is enabled by default: if node is 'unschedulable' ( is true), the pod will not schedule to it. already includes this check; is used to also enable this check when enabled.
/status in-progress
any solution here?
fluentd is scheduled on the master node, which is marked unschedulable (). Is this working as intended?
Yes, assuming fluentd is a DaemonSet pod. is nothing new; it's introduced to fix a regression in 1.10 (). As stated in DaemonSet :
If fluentd Daemon pod isn't supposed to be scheduled on the master, taints should be to the master; if instead it should be scheduled on the master, we should add more capacity to the master node.
fluentd's asking more cpu resources than 1.9 (the regression caused by masked the issue until now). Seems like this could be related to the introduction of fluentd-gcp-scaler. /assign
That's unexpected, thanks for noticing /assign Daniel, please take a look
fluentd-gcp in both 1.9 and 1.10 asks for the same amount of resources: 100m cpu request, 200Mi memory request (and 300Mi memory limit). Introducing fluentd-gcp-scaler didn't change these values. Do you mean that CPU request increased? I see it is 100m now (as it was for 1.9), what values did you see for 1.9 and now?
any updates?
The fluentd pod on the master from https://k8s- Both the and the containers request 100m cpu This amounts to 200m cpu. In v1.9, did not have any cpu request IIRC. You can double check that.
Thanks! Looks like there is a bug in the scaler, so it sets resources on all containers instead of fluentd-gcp only. Scaler fix is in , I will create a PR with version bump once this is merged.
Actually, maybe better to /reopen until testgrid becomes green.
ACK.

In progress ETA: 19/03/2018
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
https://k8s- is green now


Question: kubelet.SetupLogging() kubelet.SetupCapabilities(kcfg.AllowPrivileged) <ins> credentialprovider.SetPreferredDockercfgPath(kcfg.RootDirectory) </ins> cfg := makePodSourceConfig(kcfg) k, err := createAndInitKubelet(kcfg, cfg) if err != nil {

As per We should look for the .dockercfg in three places: ./.dockercfg, ~/.dockercfg, and $rootDir/.dockercfg
I'll tackle this.
Bunch of questions: we also add a flag that overrides the search mechanism suggested in this issue? Perhaps, ? to the user's homedir requires we use os/user [0] . The os/user package and cross-compiling don't play nice [1]. So, if we add to the search path, cross-compiled kubelet binaries will not actually be able to take advantage of it. Is this an issue? is the proper lookup order? I suggest: if provided, , , . walking through the lookups, do we stop at the first file that exists, or the first file that exists and is parsed as a valid ? [0] [1] /cc
I would absolutely love a --dockercfg option, and it would also be helpful if it logged which .dockercfg it ultimately used.
1) Seems like YAGNI to me 2) Is all of os/user broken? We will eventually want actual user support. I'm fine with dropping it for now. 3) I would say $rootDir/.dockercfg, ./.dockercfg (assuming we drop ~) 4) First successful parse sounds good to me. And can we NOT log errors if files don't exist :)
would it make sense to default to the kubelet config directory if provided rather than the working directory of the kubelet process? then no extra flag required.
What ever happened with this idea?
/cc
It never got high enough on my weekend TODO list :( I can keep telling myself that I need to do it, or we can find someone who has a bit more time than me.
I can probably get to this today.
Cool, thank you! , Mike Danese wrote:


Question: It("should exceed backoffLimit", func() { By("Creating a job") <del> job := framework.NewTestJob("fail", "backofflimit", v1.RestartPolicyNever, 1, 1, nil, 0) </del> <ins> backoff := 1 job := framework.NewTestJob("fail", "backofflimit", v1.RestartPolicyNever, 1, 1, nil, int32(backoff)) </ins> job, err := framework.CreateJob(f.ClientSet, f.Namespace.Name, job) Expect(err).NotTo(HaveOccurred()) By("Ensuring job exceed backofflimit")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: for a Job does not work on Kubernetes 1.10.0 What you expected to happen: can limit the number of time a pod is restarted when running inside a job How to reproduce it (as minimally and precisely as possible): Use this resource file: If the job was created in Kubernetes 1.9, it will soon fail: While creating the job in Kubernetes 1.10, it will be restarted infinitely: Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.0 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Kubeadm Others:
/sig scheduling
/sig job /sig lifecycle /sig apps
/sig testing
We just updated to 1.10 and seem to be affected by this, too.
I had to add activeDeadlineSeconds to all Jobs and Hooks, as workaround...
Can confirm with 1.10.1
It seems to be introduced by cc
Cause appears to be that the method gets exactly one attempt to update the status with the failed condition and if it fails due to resource conflict, it never adds the condition. The controller code seems to use requeues to track failures, but if a sync loop executes successfully, it would lose track of the failed item ever having been requeued. Looking into a fix now.
Looking at , we force an immediate resync when we see a pod that has failed which also clears the key in our queue, but that in turn makes us lose state of the number of requeues.
This bug killed my test cluster over the long weekend (with + pods). Luckily we don't yet use CronJobs in prod.
killed my cluster too :( still existing in kubernetes 1.10.2
This issue also can't be mitigated as terminated are not part of the resource quota:
This is a pretty terrible bug and it definitely still exists in v1.10.2
/priority important-soon
can you please take a look?
Looking right now...
I've opened to address the issue. Sorry for the troubles y'all.
pulled the patch into current 1.10.2 release and it works! Thanks :)
I filed (with and a test for Job with backoffLimit=1).

The test exposes some other problems. Here are two failure mode: never retries (): the code uses to see the number of retries. However, the job can fail to be synced just because of an update conflict (). That shouldn't be seen as a retry. retries more than needed (): Job controller creates more pods than needed because it doesn't realize itself has failed and exceeded backoffLimit earlier. This may be due to stale cache or that the job status failed to be updated earlier. I think Job backoffLimit is still broken.
FWIW,(assuming I'm not missing something) I can confirm on GKE 1.10.2-gke.3 (latest as of Jun 2nd) the is still broken. The following job spec endlessly creates pods:
we used the latest v1.10.3 release but backoffLimit is still broken
This issue prevents us from rolling 1.10 to all clusters
/milestone v1.11 Looks like we have a couple of PR(s) in progress and feels like we should get this fixed by the time we ship 1.11, please let me know if this fine or not.
I think that this issue should be fixed in 1.10 regardless of 1.11 as this issue causes serious cluster stability, for a fairly common scenario. Specifically for us, migration to 1.11 adds complexity and risk due to the change in the way nvidia GPUs are managed, so it will take us some time to upgrade to 1.11.
y, guess my question is ... is it fixed in master? (and hence already fixed for 1.11), if not, do we need to get this fixed in master/1.11 and backport to 1.10
Unless I'm mistaken, the objects are considered stable APIs right? (since they are under the group). If so, I also think this should be fixed in the next 1.10 point release. In general, like we would also rather to be able to use s without being forced to upgrade to 1.11.
This is a regression in functionality, and when the proper fix is determined, it should be cherry picked back to 1.10. We need it fixed in master first, however. I'm going to escalate this to sig-apps to see if we can get the right resources.

cc: This will need to merge into master, and then be cherry picked back to 1.10
PTAL
The fix is ready for review and will be cherry-picked for the next 1.10 release.
Reopening for cherry pick.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I didn't see anything in the Changelog for v1.10.4, so i'm guessing this still hasn't been fixed yet?
The fix has landed to master but not to 1.10 yet. See It will get hopefully merged before the next 1.10.x release.
Having the same issue. Waiting for update.
This fix has been cherry picked back to 1.10, and should show up in the next patch release (1.10.5).
Any ETA for 1.10.5 yet? Even a ballpark date would be fine.
I'm planning to cut 1.10.5 tomorrow, assuming all tests will be green. In case of any last minute problems it may slip by a day or two, but so far everything looks good for tomorrow.
Thankyou to all involved! I really appreciate this being rolled back into 1.10.
Given that the fix is already on 1.11 and on 1.10 branches, this issue should be closed. Thanks for fixing this!
/close
I don't believe this is fixed? Unless I'm doing something wrong.. Both resource specs have a and . This is in GKE,
I have the same bug on 1.9, with an openshift cluster: Any plan on backporting it to 1.9 as well?
you likely are encountering a different issue, as the root cause of this bug was introduced in kubernetes 1.10, not 1.9.

Also, kubernetes 1.9 is now outside of the support window, so no fixes will be backported to the upstream branch.
I also find this bug in kubernetes 1.12.1 Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} In pod failure condition,when i set parallelism 2, backofflimit 0,it will destory the last failure pod,only left one pod remain;And set parallelism 2, backofflimit 1,sometimes it keeps 3 pods,sometimes it kills the last failure pod remains 2
@ kubectl version: Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} minikube version: v0.30.0 apiVersion: batch/v1 kind: Job metadata: name: whalesay1 spec: template: spec: containers: name: whalesay image: docker/whalesay command: ["cowsa", "This is a Kubernetes Job!"] restartPolicy: Never backoffLimit: 2 NAME READY STATUS RESTARTS AGE pod/whalesay1-6lt4b 0/1 ContainerCannotRun 0 59s

pod/whalesay1-86tlz 0/1 ContainerCannotRun 0 48s pod/whalesay1-8v8nr 0/1 ContainerCannotRun 0 1m pod/whalesay1-8w9k2 0/1 ContainerCannotRun 0 1m pod/whalesay1-8xxtr 0/1 ContainerCannotRun 0 53s pod/whalesay1-dl4l8 0/1 ContainerCannotRun 0 1m pod/whalesay1-fhlhg 0/1 ContainerCannotRun 0 37s pod/whalesay1-fz8mv 0/1 ContainerCannotRun 0 26s pod/whalesay1-g2mt6 0/1 ContainerCannotRun 0 9s pod/whalesay1-hgj95 0/1 ContainerCannotRun 0 1m pod/whalesay1-hrsrd 0/1 ContainerCannotRun 0 1m pod/whalesay1-hzm42 0/1 ContainerCannotRun 0 1m pod/whalesay1-mvctw 0/1 ContainerCannotRun 0 1m pod/whalesay1-n7zq9 0/1 ContainerCannotRun 0 2m pod/whalesay1-nl89l 0/1 ContainerCannotRun 0 31s pod/whalesay1-nqcmg 0/1 ContainerCreating 0 2s pod/whalesay1-p85rm 0/1 ContainerCannotRun 0 2m pod/whalesay1-pjzmd 0/1 ContainerCannotRun 0 2m pod/whalesay1-q2k7p 0/1 ContainerCannotRun 0 43s pod/whalesay1-rhvz7 0/1 ContainerCannotRun 0 1m pod/whalesay1-rzrtg 0/1 ContainerCannotRun 0 1m pod/whalesay1-xp5tr 0/1 ContainerCannotRun 0 1m pod/whalesay1-zn2s6 0/1 ContainerCannotRun 0 1m pod/whalesay1-zq4g9 0/1

ContainerCannotRun 0 14s pod/whalesay1-ztxdv 0/1 ContainerCannotRun 0 19s NAME DESIRED SUCCESSFUL AGE 1 0 2m
Just update to something newer than 1.10.0. That version is ages old and I'm sure it is fixed within the latest 1.10.x release. (Which is also out of support so update to 1.12/1.13 ASAP.)
In version "v1.10.11" the though set to 4 for a Job's pod, the pod eviction happens for 5 times and at few times it was for 3 times. Also to note that we also have applied and Below is the version details of Kubernetes that we are using
Though we have set the backoffLimit of a Job to 4 the pod eviction is still being done more than 4 times i.e. 5 times at one of the instances its three times as well. The eviction of the pod happens for a geniune reason i.e. the disk space of the pod node exhausts when the application within the pod executes, we know the reason why that's happening. The point is that the Job's-Pod should be attempted 4 times and not 5 times. Please note that strangely we don't get to see this error in another environment where we have v1.10.05. providing the version details of that environment below There the pod eviction happens as per the stated literature of Eviction with BackOffLimit. We have no issues there. Now when we have upgraded to v1.10.11 recently [ About two days back ] we are getting the reported problem. Would appreciate if someone in this forum can advise on the root cause and a solution. Do we have any flaw with version "v1.10.5" like security loophole.
10.11 is required to avoid a critical security bug. I imagine we will need to update to fix this, but we will need to find out which of the patched versions has the fix we need. We can then roll this through the environments as a staged update.
As noted above 1.11+ are supported by the community until 1.14 is out:
I experienced the same behavior with . Can somebody confirm that this not happening with = versions?

We are experiencing the same issue with version 1.11.9-docker-1. If it makes any sense we have 2 enviroments. One is Ubuntu 18.04.2 LTS and the other is CentOS Linux release 7.5.1804. It happens only at the Ubuntu enviroment.
In meantime we updated cluster: We still experiencing the issue.
Hi everyone! This issue is for a specific problem in Kubernetes 1.10. Right now, the upstream Kubernetes project supports the following three versions: 1.13, 1.14, 1.15. If you are experiencing a similar issue under any of those three versions, then please open a new issue and provide all the details requested. As the specific bug in this issue is resolved and on an unsupported version, I'm going to lock this closed issue. Thanks!


Question: if isIndexedJob(job) { completionMode = string(*job.Spec.CompletionMode) } <del> job.Status.Conditions = append(job.Status.Conditions, *finishedCond) </del> <ins> job.Status.Conditions, _ = ensureJobConditionStatus(job.Status.Conditions, finishedCond.Type, finishedCond.Status, finishedCond.Reason, finishedCond.Message) </ins> if finishedCond.Type == batch.JobComplete { if job.Spec.Completions != nil && job.Status.Succeeded > *job.Spec.Completions { jm.recorder.Event(job, v1.EventTypeWarning, "TooManySucceededPods", "Too many succeeded pods running after completion count reached")

If the user update , duplicate condition: ! Only one Failed condition Create Job and wait it marked as Failed Update the Failed condition to False No response <details</details<details</details<details</details<details</details<details</details<details</details
It's not a bug. The is a list, so it can contain many conditions with different probe timestamps. Also, we should not update the fields.
But should we avoid the duplicate Type's conditionIn my opinionthis will confuse the user. According to kubelet, it will rewrite the 'Ready' condition in pod, and take is as managed only by kubelet. Should we did this for job Failed condition? Refer:
/sig apps
It is a bug. We should only have one condition for each type. /triage accepted
/assign
/assign
/assign
I have reproduced the issue locally and intend work on the fix
/priority backlog Since the user is manually modifying the status, this is not an inherent bug in the controller. Still worth avoiding duplicates.


Question: Subsystem:         KubeletSubsystem, Name:              ActiveConfigKey, Help:              "The config source the node is actively using. The count is always 1.", <del> DeprecatedVersion: "1.22", </del> <ins> DeprecatedVersion: "1.22.0", </ins> StabilityLevel:    metrics.ALPHA, }, []string{ConfigSourceLabelKey, ConfigUIDLabelKey, ConfigResourceVersionLabelKey, KubeletConfigKeyLabelKey},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --kubelet failed to start when specifying ,with following error: Try to start kubelet with : Kubernetes version (use ): master branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Submitted PR to fix this issue.
/triage accepted /priority important-soon


Question: }}, metav1.CreateOptions{}) framework.ExpectNoError(err, "creating CR") v, found, err := unstructured.NestedFieldNoCopy(u2.Object, "a") <del> framework.ExpectEqual(found, true, ""a" is defaulted") </del> <ins> if !found { framework.Failf("field `a` should have been defaulted in %+v", u2.Object) } </ins> framework.ExpectEqual(v, "A", ""a" is defaulted to "A"") // Deleting default for a, adding default "B" for b and waiting for the CR to get defaulted on read for b

The tests are full of assertions which compare a boolean against true or false, often without any additional explanation. When those assertions fail, the error message is useless for understanding what went wrong, often just saying: Including an explanation is better, but then we still have the distracting noise about the comparison: All of those assertions should be replace with . Such locations can be found by grepping: UPDATE (2024-08-26): in cases where such an if check is not possible (typically in combination with or ), should be used instead of . They emit only the explanation given to them, without including the noise above. The Ginkgo linter (enabled in golangci-lint) points that out. Some progress has been made in the meantime: v1.23.0: 250 v1.24.0-beta.0: 177 However, we were down to 175 at some point, so the number is going back up. Example: Some other places compare values. Those should be replaced with a comparison assertion. Example: It makes sense to split this work into individual PRs, one per OWNERS file under test/e2e. Here's a list of directories: [ ] test/e2e/apimachinery [x] test/e2e/apps: [x] test/e2e/auth [x] test/e2e/autoscaling: [x] test/e2e/cloud/gcp [x] test/e2e/cloud [x] test/e2e/common/network [x] test/e2e/common/node: [x] test/e2e/common/storage: [x] test/e2e/framework: [x] test/e2e/framework/providers/vsphere [x] test/e2e/framework/volume [x] test/e2e/instrumentation/logging [x] test/e2e/instrumentation: [x] test/e2e/kubectl [x] test/e2e/lifecycle [x] test/e2e/node: [x] test/e2e/scheduling: [x] test/e2e/storage: (prior PR: ) [x] test/e2e/testing-manifests/storage-csi [x] test/e2e/upgrades/apps: [x] test/e2e/windows: , When creating a PR, use as title. Use one commit per PR, squash when addressing feedback.

Unless noted otherwise, will work on PRs. Help may be welcome. /sig testing /help Better output for developers when tests fail.
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? Example: Some other places compare values. Those should be replaced with a comparison assertion. Example: /sig testing /help </details
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue </details
/triage accepted
I would like to take this up as my first issue
/assign
I would like to take up this issue as my first issue
Hello I am already working on this issue , can you please take some other issue ?
Okay sure ,how do I un assign myself from this issue?
I don't think you are assigned to this issue.
Oh okay
Hey as discussed in pr I am assuming that you are only working for tests under area.

also finished the work on , see
and only one PR is to be made for the correction under test/e2e/storage/, so how can we distribute the task and also make only single PR. Or can we make multiple PR's
I have a TODO list to the issue description. I'll try to keep that up-to-date with what has been done and who is working on it.
you can continue with in
what is to be changed in this case !
can you inform on which directory you are working.
Currently working on apimachinery
Nothing, just leave it as-is.
But that isn't under - before starting somewhere else, please coordinate with
test/e2e/instrumentation/logging don't have any cases like "Expect.Equal([^,], *(true|false)", so I think it should be removed from tasks to do.
I am thinking that the output should be if !ok { framework.Failf(uexec.ExitError) } in this case, am I thinking reasonable here.
isn't valid Go. The first parameter must be a format string. The code checks whether err is a : Therefore the change should be:
Why don't we just add extra explanation into calls like for easy investigation? I might miss some point.
You have to consider the complete output when that case fails. Gomega will write print the actual value and the expected value, then in addition you get the same information in the custom message. Printing just the custom failure message is shorter and more readable, in particular for a comparison of booleans. For a comparison of integers I see two solutions: ExpectEqual with a fixed message that just explains what is being compared. An if check with Failf where the message includes the actual value. The later seems more useful when the expected value is zero because that value doesn't need to be printed when its obvious.
I am getting difficulty in figuring out what should be output here, what should the appropriate output here. Also what steps you took in knowing so, it would help to improve my understanding.
The "ok" value is the result of checking for a specific key in a map. What a developer therefore wants to know when that check fails is: what was the key what's in the map Both can be included in:
can I work for the section :- test/e2e/scheduling ?

And there was only one changes needed for test/e2e/node which I did but my PR was closed as it only contains trivial edits. That was my first PR, so can you please guide me further?
/assign I'll try to coordinate the work, but won't create PRs myself. I ticked of line items which had no remaining work. feel free to start working on .
I replied there and also suggested an improvement. For those following this issue: try to include more information than before in the error message when the assertion fails. Then it is more obvious to the reviewer that the change is useful without having to read this issue. Examples:
Another tip: don't just reference this issue as explanation for the PR. Reviewers are busy, they won't necessarily invest the time and follow the link to understand what the PR is about. Instead, take the first sentence from this issue and adapt it as needed to the specific changes in the code.
can I start working on test/e2e/common/storage ?
Okay.
for test/e2e/network/netpol , there are already message associated with all of the cases, for example :- So should I just change all of them to ? And same for all the cases in test/e2e/network except these two cases : framework.ExpectEqual(netutils.I sIPv4String(internalIPs[0]) != netutils.IsIPv4String(internalIPs[1]), true) framework.ExpectEqual(netutils.IsIPv4String(p.Status.PodIPs[0].IP) != netutils.IsIPv4String(p.Status.PodIPs[1].IP), true )
Yes.
Okay then I will be working on and
It might make sense to combine them in a single PR, depending on how many changes there are - just use common sense.
IMO : this will never fail because whether err will be nil or it will exist. So, we don't need to change this code. What I had done in this case was :
Note that that is redundant (= not wrong, just not needed). I later clarified that in
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Hi! I've created a PR to address the tests in e2e/apps It is my first PR so hopefully, everything is ok. Thanks a lot!
Hi can I take test/e2e/common/node?
Yes, please do.

Have we considered / helpers that can skip the comparison output? - is noisy and makes it harder to skim tests, especially given that it's a pretty commonly required pattern.
No, we haven't. Do you find it noisy because it's two lines instead of one? I don't have a problem with that. It seems to be a common approach in unit tests and I find it easy to read, partly because it matches how production code would be written.
It becomes three, plus extra indention - but more importantly stands out in a series of expectations. This draws extra scrutiny to those lines. many unit tests using testify would use to express this because of that.
Isn't that a good thing? is bad without any or no useful extra explanation. The if check makes it obvious that a good failure message is needed. Does print less useless boilerplate code than Gomega? We could use that with Ginkgo (), but then we still need to ensure that the optional explanation is useful.
xref
Hi can I take test/e2e/auth?
for test/e2e/auth, for example : Should I just change it to ?
feel free to work on test/e2e/auth. This one particular example is a bit different: here we are comparing string values, and for those it makes sense to print the actual value because the failure message then adds information. I would leave that one as it is.
OK. test/e2e/auth doesn't need replacing. I : <img width="731" alt="Screen Shot 2022-05-30 at 9 12 59 PM" src="https://user-"There are a lot of positions that need to be modified in . I'll take test/e2e/apps.
That looks like a good candidate. Please go for it. When you need to make up new failure messages, try to think about what information may be useful to have and add that. Then the resulting PR is clearly better than the status quo.
Hi can you please assign me a directory. I want to give it a try. Thanks,
Hi I already posted a PR for e2e/apps some days ago in this issue: If it is fine I'd love to proceed with that. Just to avoid us repeating work.
sorry for not noticing the overlap sooner. Let's go with the older PR.

you can take over test/e2e/storage from There is an incomplete PR for it in You can check out that branch, fix it and submit a new PR, or start from scratch.
it looks like is not active anymore. I've remove him from the todo list and you instead for test/e2e/common/storage.
OK. Thank you. Is there an incomplete PR for .
I in and it returns 0. What should I do?
needs modifying. <img width="1440" alt="Screen Shot 2022-05-31 at 2 40 29 PM" src="https://user-"Can I work on it?
Indeed, did test/e2e/common/storage in You can work on test/e2e/windows.
OK. I'm working on it currently.
Sure, I will look into it. Thanks,
: do you want to continue with test/e2e/apimachinery?
Yes. There's an older PR:
Hi is there anything I could take over?
Hey I am not sure how updated the list is, but I am available to help. Is there any part I could start or continue?
sorry for the late reply. I had (and still have) some big changes for the E2E tests going and had put this on the backseat for a while. If you are interested, how about doing test/e2e/apimachinery and test/e2e/autoscaling? Some work was done on the latter, but there are still several calls (again?).
Sure, I can take that!
Hey I'd also like to contribute on this issue. Is there anything I could take over?
feel free to take my part:
Perfect, then I will take part
Hi :smile: , I would like to contribute to this issue, there is anything that I could do?
I was reluctant to start new work on this because I had one big PR pending that touched a lot of test code (PR ), but that has been merged. you are taking test/e2e/autoscaling, right? test/e2e/apimachinery? you can do test/e2e/apps. That leave test/e2e_node in case someone wants to continue.
yes e2e/autoscaling. I had some trouble to get the tests going initially, but seems fine now!
Hi, Can I do something to contribute to this issue?

you could work on test/e2e_node.
Ok, I will start work on test/e2e_node thanks!
Hi , I would like to contribute. Is there anything I could take over?
thanks for your interest, but right now everything seems to be covered.
Hi Thanks. I am searching for "good-first-issue", feel free to assign one to me. I am exploring api-machinery, but happy to contribute in other sig as well.
maybe you can take a look at if you have some time :)
Hi , There is anything that I could do to contribute with this issue?
Hi , I was looking and found the test/e2e/apimachinery without doing it and only then did I see that it was already assigned if this is a problem just refuse my pr and if you have something else to do let me know.
The remaining work will probably finished in , so there isn't anything else to do. If someone wants to investigate writing a linter for this, then this would be a cool science project - not easy, though. I would start with looking at for this. It has the advantage that it is supported indirectly in golangci-lint ().
Hello I would like to try to do a liter for it. It will be a cool project whit a lot of opportunities to learn. There will be a separate issue for the linter or we continue the discussion here?
Let's continue here.
I ended up creating a separate issue for linter ideas: let's continue there.
Ok!
New tests have been again with this anti-pattern since the last time that I checked. It's clear that without a linter, we'll not get rid of it. So let's focus on that first.
is this issue still open
Yes, this is still open. Current status as of v1.28.0-rc.0: Two calls are in , the rest are in . We have such a linter now, with ginkgolinter in golangci-lint and (soon) the ability to prevent usage of , so it makes sense to continue with removing those calls. do you want to prepare one PR for all remaining calls? I think one PR is enough, no need to split it up.
also has it, which was intentional. I'm undecided whether that should be replaced.

Several of those calls are being touched in , let's wait for that PR to be merged before doing any further work.
no longer finds anything. We might be done... :tada: But not quite. What we have left now is checking a boolean via or , which has the same problem. In some cases, there is an extra message (which is still noisy, see ), in other cases there isn't even that (which makes the failure message fairly useless). We can and should replace some of those calls with the pattern.
is almost resolved. My work-in-progress branch forbids the usage of in favor of the new or plain Go. In some cases, moving the Go code that returns true/false into a matcher might be the best solution because that allows dumping the full actual value that was being checked. Here are the remaining issues: <details</detailsDoes anybody want to work on those?
I would love to!
: please go ahead. Assign PRs to me when you create them. Let's start with a few of those calls initially, do a round of reviews, then continue. /assign
You will have to start with my branch, current master doesn't have gomega with thew new API yet.
I'm starting new to kubernetes. Is there anything easy that I could do ?
Hey Is there anything left I can help you with in this issue ?
Thanks for your interest. There's not much to be done at the moment, though.
Should this be closed?
Oh wow! Looks like we are indeed done. Thanks everyone. /close
Closing this issue. <detailsIn response to : Oh wow! Looks like we are indeed done. Thanks everyone. </details


Question: export KUBE_GCE_ZONE=${E2E_ZONE} export KUBE_GCE_NETWORK=${E2E_NETWORK} export KUBE_GCE_INSTANCE_PREFIX=${KUBE_GCE_INSTANCE_PREFIX:-} <del> export KUBE_GCS_STAGING_PATH_SUFFIX=${KUBE_GCS_STAGING_PATH_SUFFIX:-} </del> export KUBE_GCE_NODE_PROJECT=${KUBE_GCE_NODE_PROJECT:-} export KUBE_GCE_NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-} export KUBE_OS_DISTRIBUTION=${KUBE_OS_DISTRIBUTION:-}

We seem to have an occasional consistency issue between the client and server versions on our Jenkins GCE e2e project. Build 6295 failed this morning because the client version was newer than the server version: The server version matches the build was was immediately running in 6294, which was aborted early. I wonder if (somehow) the cleanup didn't work properly? (Which is a bit worrying if so.) Here's the version from 6294: A short while later, build 6297 failed, this time with the client version older than the server version: This one makes a little bit more sense: there was a new build published between downloading the client/test tarballs and setting up the cluster. I'm guessing that somewhere in the cluster setup scripts the latest CI release file was read again, rather than using the version read for the client/test tarballs. (I haven't yet worked out where exactly it's read, but it should be an easier fix.) cc
A side issue: we should probably figure out why the build git client is being marked dirty. Since we're checking out the head version from GitHub, it should ideally be a clean client.
The -dirty bit is now fixed by .
This is caused by different Jenkins jobs uploading to the same GCS bucket and clobbering each other. In we see: Over in
This also explains the issues seen in ; the job is also running in the same project.
Ah, good sleuthing. We avoided this issue in the PR builds by setting (used ), but maybe we should always be setting to include the Jenkins job name (and optionally executor/node number).
The original idea was to run every e2e job in it's own project, to avoid these sorts of name clashes. But we seem to have regressed and now have many jobs sharing the same projects. Is project-per-job not a better approach? It also helps with resource leaks and the likes. Just a thought.
I'd argue this is actually uncovering a (dev) deployment bug; I should be able to deploy different versions of Kubernetes to the same project. I know you and I disagree (and probably always will) on the strategy here.

:smile:
fwiw I think I agree with that if project-per-job avoids this issue and helps identify resource leaks, it's a pragmatic near-term improvement in the state of the world that would result in more benefit than the cost of changing all the everythings to support multi-job-in-project. Re: devs deploying different versions of k8s to the same project, P3 feature request?
I was just about to suggest the same thing. I will file a separate issue, and fix this in the short term an easier way. is out to make it easier to isolate a run to a project.
Also it seems like a reasonable thing (higher than P2) to want to deploy multiple clusters to the same project safely in prod, not just dev. Disagree?
Multiple clusters in prod using well-defined build versions such as v1.1.4 = P0. Multiple clusters in prod using our cooked-up test-building system = non-goal?
SGTM; this is prod code, though, not test code; it's used every time we call , regardless of whether it's a CI build or released version.
FYI: kubernetes-e2e-gce-reboot/ has one reboot test failed due to kube-proxy image change. But that build doesn't include pr yet.
looks like it. That run used version , but the first build with was .


Question: "type": "pd-ssd", "zone": cloudZone, }, <del> "1.5Gi", "2Gi", </del> <ins> "1.5G", "2G", </ins> func(volume *v1.PersistentVolume) error { return checkGCEPD(volume, "pd-ssd") },

GCE PDs are allocated in chunks of GBs (not GiB) but function incorrectly creates volume in chunks of GiB - cc /sig storage
/kind bug
/assign
/priority important-soon
Hey I submit a pr for this issue. Your suggestions are very important. Thanks.


Question: results = append(results, groupVersion.WithResource(apiResource.Name)) } } <del> return results, nil </del> <ins> return results, utilerrors.NewAggregate(allErrs) </ins> }

I have been having the kube-controller-manager (running in a container) crash quite frequently with this error: I cant fathom what seems to cause it. It seems to happen consistently for a while (CrashLoopBackoff occurs) and then fixes itself after an undetermined amount of time. Kubectl version: Docker version: Networking: Calico (with the calico k8s policy agent) Please do let me know if there is any other information that I can supply. I appreciate I haven't given much to go off of. I am not sure what the steps to reproduce are...
cc
It looks like a bug. I'll take a look. Thanks for reporting it.
Weirdly, the description showed up in the email github sent me but it appears absent on the github issue?
The description was there when I read it but disappeared now.
Well thats very odd! If you have it on your email, could you post it on here please (and I will update the description)? I unfortunately haven't got the error message to hand now...
I took the liberty of just editing the OP.
thanks a lot! Not sure what happened there!
could you paste the returned output at "server-ip:port/api" and "server-ip:port/apis"?
Sure.
Thanks "" has an empty preferred version and that's causing the error you reported. Is "" created via the thirdparty resource api? ()
aha! Good spot! Yes, it was created via that API. This is the Yaml file I used to create it: Here is the output of
Thanks for the confirmation Your configuration is correct. The preferred version is left empty by the thirdparty resource controller, while our discovery client has an assumption that the preferred version won't be empty. I'll patch the discovery client to tolerate an empty preferred version, and will discuss with on if we should make a default preferred version for a thirdparty resource.
Thanks a lot Great job on working it out :)
Sent to fix.
What would be a workaround for this before the release, how can one specify the preferredVersion?
AFAIK, there is no workaround. There is no way to specify the preferredVersion for the thirdPartyResource.

I have a quick workaround to fix this : 1) Change the kubernetes to reflect the following : from : thirdpartyresources=true to : thirdpartyresources=false 2) Then restart 'kubelet' service OR reboot the K8S Master server systemctl restart kubelet 3) Revert back the changes done to kube- from : thirdpartyresources=false to : thirdpartyresources=true 4) Verify whether the above change on the above manifest file has been automatically picked up by the kubernetes API ps aux | grep -i thirdpartyresources 5) Finally you may restart kubelet service systemctl restart kubelet 6) Now verify whether the 'kube-controller-manager' is in a 'Running' state kubectl get pods --all-namespaces | grep -i controller-manager Hope this fix works for you guys as well !!


Question: roundedVal: int64(1048576000), }, { <del> name:        "round overflowed quantity to int64", resource:    resource.MustParse("73786976299133170k"), </del> <ins> name:       "round decimal to KiB", resource:   resource.MustParse("1.2Gi"), roundedVal: int64(1258292), }, { name:       "round big value to KiB", resource:   resource.MustParse("8191Pi"), roundedVal: int64(9006099743113216), }, { name:        "round quantity to KiB that would lead to an int64 overflow", resource:    resource.MustParse("8192Pi"), </ins> roundedVal:  int64(0), expectError: true, },

After upgrading our clusters to Kubernetes 1.19, the provisioning of PersistentVolumes started failing with the following Events: The value matches the actual storage request of 3.2Gi. It doesn't seem special in any way, not too big, not too small, and it always worked until 1.19. We look into the issue with and we believe the PR introduced a change that broke the PersistentVolume provisioning even when it doesn’t overflow int64. Amounts of resource in Kubernetes are represented with the . In the PersistentVolume drivers, there are some functions that manipulate storage request in order to allocate a disk with the right size. Before the mentioned PR, those quantities were retrieved by calling which relies on to return the value as an int64. This worked probably fine in most of the cases, but it could happen that the represented value would overflow an int64, and in this case the result of would be wrong. The function’s description states that it’s the caller’s responsibility to check this: This is likely what the PR wanted to handle. The change replaced all calls with which has a different signature. From where it’s used now, you would think that the second return value signals an overflow: : This is probably where the error we’re seeing is generated. I believe that the signature of was misinterpreted. From my understanding of its description, returning doesn’t mean we’re overflowing . It only means that internally, the quantity is represented as a decimal amount: Internally, depending on their values, quantities are stored/represented in : for DecimalExponent (e.g. ) or DecimalSI (e.g. ) for BinarySI that don’t have a decimal part but a unit (simplified) (e.g. ) for all the rest With the PR, Kubernetes now fails to provision volumes that have a storage request represented as . Many of the storage requests are represented this way and still perfectly valid in our opinion. I would have expected the provisioning to continue to work as those requests seemed valid and were always accepted by our underlying cloud-provider in earlier version of Kubernetes.

Create a StatefulSet with a PersistentVolumeClaim that uses vSphere storage: Kubernetes version (use ): 1.19.7 Cloud provider or hardware configuration: vSphere, but I think it impacts other cloud-provider that were changed in the PR: /sig storage /kind bug /cc Possibly related to
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think the only reason we do not use the quantity library for scaling and why we tried to roll our own rounding code is because when scaling stuff in AsScaledInt64(3) or ScaledValue(3), the library assumes decimal SI (M, 1000), not binary SI units (Mi, 1024): You are right, our self-rolled rounding code doesn't handle values with millibyte or higher precision like 3.2Gi. Our code uses AsInt64 and then performs its own scaling. If AsInt64 returns false, the int64 returned is not in bytes (it must be in millibytes or higher precision), so our rounding code won't work, so it returns that " is too great, overflows int64". In 1.18, we used Value() to round to bytes but we didn't check it against ScaleValue as the signature suggested. So the PR in question fixed cases where Value() overflowed but inadvertently broke your case. The correct way to round, which the resource library's rounding does, is to drop the useless precision and round up. , but as I said it doesn't seem to handle binary SI quantities correctly. For decimal SI quantities, we should rely on the resource library's AsScaledInt64 and be done with it. (We could also use Value() and ScaledValue as the signature suggests but calling AsScaledInt64 directly seems simpler). Like this: For binary SI quantities, we need to do something fancier involving CanonicalizeBytes I think, I'm not sure yet.
I guess for binary SI we can go back to using Value but actually check the overflow this time.

Hi Thanks for the quick feedback on the problem's description already. Indeed, the Scaled() functions don't handle BinarySI and that's why I quickly discarded the idea of using them. Since this is about storage, I suppose most people are using this format. I just finished wrapping up a PR that I hope address both the issue we are facing, but also what was meant to be fixed in the PR your reviewed back then. It doesn't make any difference between the 3 formats however.
Yeah I commented within seconds of your PR. I glanced at it , will review more soon, but it makes sense to me to just use VAlue and check ofr overflow for both decimal SI and binary SI rather than using Scaled() for one and not the other :+1:
I think it means both. i.e - when incoming value is decimal or if value is so big that type can't store it as it switches to decimal. So while my PR fixed later case, it broke the former. :/


Question: "runtime" v1 "k8s.io/api/core/v1" <ins> "k8s.io/component-helpers/scheduling/corev1" </ins> "k8s.io/klog/v2" v1helper "k8s.io/kubernetes/pkg/apis/core/v1/helper" <ins> "k8s.io/kubernetes/pkg/kubelet/types" </ins> "k8s.io/kubernetes/pkg/scheduler" schedulerframework "k8s.io/kubernetes/pkg/scheduler/framework" <ins> "k8s.io/kubernetes/pkg/scheduler/framework/plugins/tainttoleration" </ins> ) type getNodeAnyWayFuncType func() (*v1.Node, error)

The method doesn't check taints and tolerations. This is problematic in 2 scenarios: If the users bypass scheduler by setting a value in If the node restarts. The node would try to restart scheduled pods once it comes back up, even if it's not ready (has a not-ready taint). Kubelet to check taints and tolerations. a taint in a node. a Pod with set to the Node above, without a matching toleration. /sig node /sig scheduling
this reminds me of your kubelet sync PR ...
I don't think this is a syncing issue. The logic doesn't check taints at all.
scenario 1 is pretty bad, i will let the #sig-node slack know about this.
It doesn't sound like a syncing issue, but we seem to have multiple issues with pod admission/management on restart currently
I'm aware of that issue. This should be unrelated.
I would like to address this. /assign
I think it would be great to proceed after merging ref:
Please add me as reviewer
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale PR in review:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: "k8s.io/klog/v2" v1 "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/api/resource" </ins> "k8s.io/apimachinery/pkg/util/sets" clientset "k8s.io/client-go/kubernetes" restclient "k8s.io/client-go/rest"

A kubernetes cluster is normally constructed from a variety of machine templates. Different instance groups may have different CPU, memory capacity. They may be equipped with specialized accelerators, network devices. I think we should have an extensible way for users to custom node configuration to simulate real production cluster states. Currently, nodelabel and nodetaints are supported. I am consider to accept a template with more configuration like resources so we won't grow option list in the future. Without this feature, it's hard for users to simulate heterogeneous cluster.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scalability


Question: done } <ins> # Saves prepulled Windows Docker images list to ${WINDOWS_LOGS_DIR}docker_images.log # on node $1. function export-windows-docker-images-list() { local -r node="${1}" local -r powershell_cmd="powershell.exe -Command "$logs=$(docker image list); $logs | Out-File -FilePath '${WINDOWS_LOGS_DIR}docker_images.log'"" # Retry up to 3 times to allow ssh keys to be properly propagated and # stored. for retry in {1..3}; do if gcloud compute ssh --project "${PROJECT}" --zone "${ZONE}" "${node}"  --command "$powershell_cmd"; then break else sleep 10 fi done } </ins> # Saves log files from diagnostics tool.(https://github.com/GoogleCloudPlatform/compute-image-tools/tree/master/cli_tools/diagnostics) function save-windows-logs-via-diagnostics-tool() { local node="${1}"

When a test run completes it would be useful to have a list of all the test containers that were pulled onto the node - see for the motivation. Let's add a call to at - just after seems like it should work fine. We can dump the output to a file and gather it from each node just like we do for
/assign Yang, we should also consider adding to the log gathering that the GCE diagnostics tool does :)
/sig windows
/reopen in the of our on 03-22 I don't see in the artifacts for the Windows node. Can you check on this? Did we miss a step or is something failing?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Oh, missed to add the new log file to collector list.
Added the fix:


Question: kubeserver "k8s.io/kubernetes/pkg/kubeapiserver/server" ) <del> // NewSecureServingOptions gives default values for the kube-apiserver and federation-apiserver which are not the options wanted by </del> <ins> // NewSecureServingOptions gives default values for the kube-apiserver which are not the options wanted by </ins> // "normal" API servers running on the platform func NewSecureServingOptions() *genericoptions.SecureServingOptions { return &genericoptions.SecureServingOptions{

Currently as authentication and authorization are both standard config for all clusters, the kubeapiserver insecure http handler is unsecure, which means the cluster is vulnerable to the attack. So it should be deprecated, even removed totally.
/sig api-machinery
I think removing the insecure option is a good idea. Can you send an email the sig-apimachinery mailing list with a proposed timeline? I think this should be like other GA features and have a one year clock.
Sure, I will send it ASAP.
:+1:


Question: containerList, _ := runtimeService.ListContainers(nil) for _, c := range containerList { if _, exists := podSandboxMap[c.PodSandboxId]; !exists { <del> return nil, fmt.Errorf("no PodsandBox found with Id '%s'", c.PodSandboxId) </del> <ins> return nil, fmt.Errorf("no PodsandBox found with Id '%s' for container with ID '%s' and Name '%s'", c.PodSandboxId, c.Id, c.Metadata.Name) </ins> } containerMap.Add(podSandboxMap[c.PodSandboxId], c.Metadata.Name, c.Id) }

Add more message for no PodSandbox container when kubelet start. When error return, We need more message to solve it, contains container name and container id.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign