Question: tags = ["automanaged"], deps = [ "//pkg/auth/user:go_default_library", <ins> "//pkg/util/cert:go_default_library", "//pkg/util/net:go_default_library", </ins> "//vendor:github.com/golang/glog", "//vendor:github.com/rackspace/gophercloud", "//vendor:github.com/rackspace/gophercloud/openstack",

Hi, I tried to integrate the keystone auth plugin (which would be perfect for my needs) into the cluster I'm currently setting up. However, I can't get the connection working since our keystone installation is currently only reachable over http As far as I can see in , there is no option to deactivate this check. I think this should be an optional parameter on startup, similar to defaulting to the check being activated.
I don't think we should enable username/password checking over http.
I agree that we are loosing a security layer here, but you can't guarantee you got the password from client over https, so it might be intercepted at this point anyways. More general, I think the proper way to authenticate to Kubernetes with the Keystone plugin using tokens since they will limited lifetime restricted on keystone only which is a big deal regarding keystone likely being bound to a LDAP and thus the password being valid for other services. So I don't want to check username/password directly. I'm just starting out with the Keystone plugin, so I want to point this out in general, if token auth is supported, the better. Couldn't try it in praxis for now since I can't get the auth URL running.
Assigning to to add a priority label.
Setting the priority low until we have a sense of how many users have this restriction (keystone only reachable over http).
I ran into this same issue with trying to deploy kubernetes with devstack in the same vm and wanted to test integration and do some development. I think there are a significant number of developers in this same boat. http is secure enough since it doesn't leave the developers vm and it isn't intended for production. The use case is the same as:
I'd vastly prefer allowing specifying the CA for the keystone server over allowing http auth calls carrying usernames and passwords
I don't disagree. :) But, I also think Kubernetes shouldn't handle the usernames/passwords at all. In http mode, sending a token is much less risky then a username/password since they expire relatively quickly. See this issue for details:
I agree tokens would be preferable, but I wouldn't expect credentials of any kind sent over http to remote servers
Agreed. No one in their right mind should be passing creds or bearer tokens over http over a network.

I'm just interested in being able to easily deploy a k8s in DevStack with a simple plugin, where that k8s is hooked into the deployed keystone so that we can start developing things in OpenStack that depend on k8s. This gives developers a smooth/easy experience to then focus on doing the development stuff and not have to mess with setting up a CA/k8s, which will prevent some of them from contributing to k8s/OpenStack. The idea is simple for developers, secure for operators. Totally separate use cases.
It should be as simple as telling k8s to trust the keystone cert as a CA (assuming the self-signed cert generated by keystone is correctly marked as a CA, which it should be, since it is signing for itself). I've seen too many production installations running with http or untrusted TLS settings to believe that only developers will use this option if it's made available.
I didn't think Devstack could easily deploy with SSL enabled, but it looks like you can: So http support might not be as important as I thought. It could be relegated to a documentation issue for deploying Devstack in this mode when deploying with Kubernetes. We'd still need a way to specify the root CA though for Keystone.
So, I played around with this a little. For my current non devstack based system, I've deployed kubernetes as a cluster running on my laptop deployed as a set of pods via /etc/kubernetes/manifests. Only kubelet is native and all the rest of the services are containers direct from google. Its working well, but since I have 3 api server containers, and get the containers direct from google, I'd have to jump through a bunch of hoops to add the keystone ssl ca cert to each of the containers. I still think an option to specify the ssl cert for keystone would be useful rather then having to merge it into the system's chain of trust.
That's how I'd want to handle this as well


Question: package create import ( <ins> "strconv" </ins> "testing" rbac "k8s.io/api/rbac/v1"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which test(s) are failing: Since when has it been failing: since upgrading golang: Reason for failure: Since Go 1.15, type casting from int to string using is not allowed, output: Anything else we need to know: related links: golang/go golang/go
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the fix should be straight forward, so I can create a pull request to fix this. /sig testing /sig cli /assign
/assign


Question: } type rbd struct { <del> volName  string podUID   types.UID mon      []string pool     string id       string image    string keyring  string secret   string </del> <ins> volName string podUID  types.UID // capitalized so they can be exported in persistRBD() Mon      []string Pool     string Id       string Image    string Keyring  string Secret   string </ins> fsType   string <del> readOnly bool </del> <ins> ReadOnly bool </ins> plugin   *rbdPlugin mounter  mount.Interface // Utility interface that provides API calls to the provider to attach/detach disks.

etcd cluster backing apiserver died spontaneously, after restart apiserver pulled up the pods on different machines. Node storage was backed by ext4 over rbd, which became mounted on several nodes at once. That could have possibly led to data corruption. I guess that the proper way to mount rbd volume is to take a lock on it first, which will at least fail if it's locked somewhere else, requiring manual intervention, but not destroying data. The lock should be optional, in case a proper cluster fs is mounted over the rbd. In this case there was clearly no need for accessing the data from several pods so ext4 is deemed a proper fs.
, Vladimir Pouzanov < :
want to get some detail. How is your pod-to-rbd mapping like? Since rbd image is a block device, it can be used by one pod at a time. I am not sure if locking is a proper way to go - if the pod holds a lock but dies in the middle of mounting, the rbd image (or any other volume type) is then excluded from access. In such case, a high level coordination at apiserver is a better place to resolve this problem.
Here's my rc: I guess it's better to have a lock to be cleaned up manually rather than to have non-cluster fs to be mounted in several locations at once?
thanks, this pod looks good to me. what would cause multiple nodes run the same pod?
I don't know the exact reason of why kubelet didn't unmount the old fs, but after apiserver and scheduler were restarted pod got allocated on different node.
It is a scary thing to see etcd down, it is meant to be reliable...
Unfortunately I don't have long-term logging in test cluster so I have no idea why it actually died. but using kubedns as primary node resolver was clearly a bad idea, once it died, nodes stuck with no resolver at all.
your thought on HA?
Wrong Tim St. Clair?
- you HAVE to be ready to handle network partitions - the old node might still be alive and even using the volume, but the apiserver can't see the node, so it thinks it deletes the pod and starts it elsewhere. If RBD corrupts itslef in this case, it is not useful and we should strip it from the codebase.

, Huamin Chen wrote:
that makes sense. rbd doesn't corrupt itself. This is inconsistency happens when you have two pods mounting the same rbd image. How does GCE PD handle/prevent this case?
It doesn't allow more than one read-write attachment. Multiple read-only attachments is OK. Better to fail than to allow corruption. , Huamin Chen wrote:
That's neat. I have a lock fencing for rbd, testing now.


Question: if err != nil { t.Fatalf("Unexpected error: %v", err) } <del> c.Convert(&Foo{}, &Bar{}, 0, nil) </del> <ins> err = c.Convert(&Foo{}, &Bar{}, AllowDifferentFieldTypeNames, nil) if err != nil { t.Fatalf("Unexpected error: %v", err) } </ins> } func TestConverter_meta(t *testing.T) {

In Go you cannot take the address of a because the map implementation hides that from you. Our conversion library assumes that any reflected map value can have Addr() taken, which causes a panic when you try to do the following for PodInfo conversion: The conversion code tries to call Addr() on the reflect.Value (the map's value, ContainerStatus value object) which fails because it cannot have an address. I worked around it by registering a conversion function on *PodInfo and then manually converting the map.
Oh you mean the reflection code can't addr that? ok sounds like a bug. Non-reflection code can certainly do it:
That playground is taking the address of a value before it's in the struct. You can't take the address of m ----- Original Message -----
Right, because go does magical c++ style references there. But I thought the map copying code handled that case specially already?
I think I see what you're talking about. I'll make a test.
I repro'd. Will fix.
It was actually the source value that caused the trouble. Thanks for the report!
Sorry, should have been clearer. ----- Original Message -----
Note that this map violates our API conventions. It should be a list of named subobjects.
Yeah this field has always been problematic. It's still a bug regardless-- could have triggered copying other maps.


Question: // Move the old manifest into the old-manifests directory if err := pathMgr.MoveFile(currentManifestPath, backupManifestPath); err != nil { <del> return rollbackOldManifests(recoverManifests, err, pathMgr) </del> <ins> return rollbackOldManifests(recoverManifests, err, pathMgr, recoverEtcd) </ins> } // Move the new manifest into the manifests directory if err := pathMgr.MoveFile(newManifestPath, currentManifestPath); err != nil { <del> return rollbackOldManifests(recoverManifests, err, pathMgr) </del> <ins> return rollbackOldManifests(recoverManifests, err, pathMgr, recoverEtcd) </ins> } fmt.Printf("[upgrade/staticpods] Moved upgraded manifest to %q and backed up old manifest to %qn", currentManifestPath, backupManifestPath)

kubeadm etcd upgrade modifying recovery steps /assign /sig cluster-lifecycle /kind bug


Question: } func (rc *resourceMetricsCollector) collectPodCPUMetrics(ch chan<- metrics.Metric, pod summary.PodStats) { <del> if pod.CPU == nil { </del> <ins> if pod.CPU == nil || pod.CPU.UsageCoreNanoSeconds == nil { </ins> return }

kubelet panic when accessing metrics/resource endpoint kubelet log kubelet works fine access metrics/resource endpoint of kubelet No response <details</details<details</details<details</details<details</details<detailsINFO[2022-07-14 05:36:21.474173892Z] Starting CRI-O, version: 1.23.0, git: (dirty) Version: 1.23.0 GitCommit: GitTreeState: dirty BuildDate: 1980-01-01T00:00:00Z GoVersion: go1.17.4 Compiler: gc Platform: linux/amd64 Linkmode: static BuildTags: apparmor, excludegraphdriverdevicemapper, seccomp, selinux SeccompEnabled: true AppArmorEnabled: true </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node


Question: } func (i *internalContainerLifecycleImpl) PreStartContainer(pod *v1.Pod, container *v1.Container, containerID string) error { <del> if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.CPUManager) { </del> <ins> if i.cpuManager != nil { </ins> err := i.cpuManager.AddContainer(pod, container, containerID) if err != nil { return err

Recently merged PR () broke clusters brought up with only GA APIs+features. See conformance test at /sig node /priority critical-urgent /kind bug /milestone v1.18 /assign


Question: utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/tools/record" apiv1resource "k8s.io/kubernetes/pkg/api/v1/resource" <ins> v1helper "k8s.io/kubernetes/pkg/apis/core/v1/helper" </ins> v1qos "k8s.io/kubernetes/pkg/apis/core/v1/helper/qos" "k8s.io/kubernetes/pkg/features" statsapi "k8s.io/kubernetes/pkg/kubelet/apis/stats/v1alpha1"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug Description: According to the preemption's doc, we'll take priority over Qos; but in , the higher priority BestEffort pod maybe rejected because of memory pressure. If TaintNodesByCondition enabled, the pod need memory pressure toleration. /cc , correct me if any mis-understanding.
/sig scheduling /sig node
If the node has memory pressure, it rejects all pods that do not tolerate it. This is orthogonal to QoS and Priority.
Preemption and node evictions do not work with the same exact metrics. Preemption is unaware of usage, while node eviction logic has pod and node resource usage information. Node eviction takes usage into account both when admitting a new pod and when evicting an existing pod. has recently sent out to update node eviction doc.
, but will also reject it if pod tolerate it :(
, please refer to for what I mean :). BTW, you need to configure out and predicates, and enable alpha feature. /cc
I'm not familiar with the Preemption/Priority design, so I don't know where is preemption happening (Node or control plane). But despite that I wouldn't read what you wrote as inconsistency. Priority takes precedence over QoS when it can. But if some QoS Pods are not allowed to run somewhere it's not an issue with preemption/priority or QoS. It's a separate predicate. Slight OT, but related to your proposal, I think we should go another way. We should prevent BE pods to get MemoryPressure toleration. Having this can result with bad user experience, as Pod will get scheduled on a Node, and then immediately killed, because Kubelet ignores tolerations. Or we can teach kubelet respect tolerations for NC created taints. I'd consider it a blocker for beta of TaintNodeByCondition.
Yes; but if BE pod has higher priority, the other pods with lower priority will be evicted by kubelet. This maybe better, no idea on which pod will be BE but with higher priority :) , do you think we should reject BE pod with MemoryPressure toleration, even it has a higher priority?

I would like to mention several points here: of all, I should say that a high priority BE pod is a result of bad configuration and we should not optimize for it, but given that it is not rejected by the system, we must handle it. BE pod will never cause preemption due to lack of resources, because a BE pod needs zero resources from scheduler's point of view. of point , I think we should not worry about a high priority BE pod getting rejected by kubelet or causing other pods to be evicted from a node. We should document the system behavior and have clear guidelines and point out the problems that could be caused by bad configurations. After that we should leave it to the user to decide if a configuration makes sense for their use-case.
Overall, I agree :). Personally, I'd like to keep components behavior align: reject by admission/kubelet or accept by them. Currently, apiserver/scheduler accept and handle it and kubelet will reject it.
Kubelet does not reject a BE pod with high priority either.
I mean BE pod with Memory Pressure toleration :)
I see. The pod may still be scheduled without any problem if another node in the cluster is found that does not have memory pressure. So, I wouldn't call this inconsistent behavior of components. Anyway, I leave it up to SIG node to decide whehter the current behavior should be changed.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: for i, test := range tests { t.Run(test.name, func(t *testing.T) { context := initTestMaster(t, "taint-based-evictions", admission) <del> defer cleanupTest(t, context) </del> // Build clientset and informers for controllers. externalClientset := kubernetes.NewForConfigOrDie(&restclient.Config{

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: TaintBasedEvictions Since when has it been failing: probably since it was in this PR? Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs -- TaintBasedEvictions probably since it was in this PR? Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17 </details
cc.
Looking into this. We moved it to integration because it was flaking in e2e too
/assign damemi
Not sure yet, but I do see this in the logs of the failing tests: which, if I understand this, would remove all the taints and mark nodes as reachable (which would explain why the test is failing waiting for node taints: ). But I need to figure out why all the nodes are becoming unavailable
is that a full fix for this flake or just partial? I still saw it failed yesterday so just checking... :)
I think I also run into this on 13th Oct: Full logs:
Now gets merged, which might fix the flake. Let's keep an eye on this.
reported it still flaked: Can you continue looking into this?
I'll keep trying, from my previous debugging it seems that in some cases the test cluster is entering full disruption mode and removing all of the taints from the nodes. I know we specifically made changes to the test (using multiple nodes) to ensure it wouldn't become fully disrupted, do you have any idea what could still be causing that?
There is some flag controlling how the dispution thing is managed internally, maybe you can check how a node lifecycle manager is constructed.

The test still flakes after we merge , we still need to investigate the cause ,
/assign
Another flake:
These flakes are consistently (always) timeouts waiting on node taints:
I'm working on this. Will send out a PR soon.
/reopen We see another flake since got merged: However, don't panic :) The original "Entering master disruption mode" cause has been fixed - you won't see that message in the above log. This time it's a smaller issue. I will explain it in another PR.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen The job is still flaky
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close But for another test :)
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: time.Sleep(tt.sleep) actualErr := ctx.Err() actualCause := context.Cause(ctx) <del> assert.Equal(t, tt.expectErr, actualErr, "ctx.Err()") assert.Equal(t, tt.expectCause, actualCause, "context.Cause()") </del> <ins> ci, _ := os.LookupEnv("CI") switch strings.ToLower(ci) { case "yes", "true", "1": // Skip. default: assert.Equal(t, tt.expectErr, actualErr, "ctx.Err()") assert.Equal(t, tt.expectCause, actualCause, "context.Cause()") } </ins> }) } }

We observed this recently. Steps to reproduce: clone kubernetes test ./test/utils/ktesting -c -race /root/go/bin/stress - TestCause /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing even after , we see this flakiness. This is easily reproducible in our local machines.
See Yes, it only reduced the flaking rate IMO. It still flakes in /cc
/milestone v1.30 Does fix the flake?
This still occurs after The change above is only skipping the Test for cases where . But this failure is occurring at a different stage while comparing expectErr and actualErr at Hence this change would fix it. Can you PTAL?


Question: "github.com/onsi/gomega" ) <ins> type watchEventConfig struct { framework           *framework.Framework resourceVersion     string w                   *cache.ListWatch jobName             string watchEvent          watch.EventType extJob              *batchv1.Job updatedMetadataType string updatedKey          string updatedValue        string } </ins> var _ = SIGDescribe("Job", func() { f := framework.NewDefaultFramework("job") parallelism := int32(2)

[x] Test approval issue : [x] Test PR : - Needed an update on the handling of anotations 27 March 2022 [x] Two weeks soak end date : 6 April 2022 [x] Test promotion PR : [x] Update Test PR: 15 April 2022 [x] Two weeks soak end date : 4 May 2022 with start of 1.25 [x] Update Test Promotion: According to these APIsnoop queries, there are still four remaining Job endpoints which are untested. Using a number of existing e2e test practices a new has been created for this job lifecycle test. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen for tracking e2e testgrid and promotion PR
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return r } <del> // URL returns the current working URL. </del> <ins> // Error returns any error encountered constructing the request, if any. func (r *Request) Error() error { return r.err } // URL returns the current working URL. Check the result of Error() to ensure // that the returned URL is valid. </ins> func (r *Request) URL() *url.URL { p := r.pathPrefix if r.namespaceSet && len(r.namespace) > 0 {

When constructing a , e.g. to stream I/O from a pod: if ends up set, will still return a result without indicating that something went wrong, resulting in a misleading error further down the line. (In my case, I wasn’t providing the correct scheme; the parameters therefore couldn’t be encoded, and the SPDY upgrade failed with an error saying that at least one of stdin, stdout or stderr had to be provided.) Other operations on fails as appropriate if is not ; I would expect to do so too. I realise that means a backwards-incompatible change. Another solution could be to provide a (same as on the response). See above, with any scheme other than client-go’s. No response <details</detailsN/A N/A N/A N/A N/A
this is the same as standard library, the url is not validated when creating the request
is it really the same though? In the case which led to my filing this issue, the call to results in an error, which is recorded in , but the caller has no way to know that. The main issue I have with this is that in most code paths, does end up being returned, e.g. calls to etc. But callers using only have no way of knowing that the URL they get is unusable.
I didn't understand that from the description sorry I see what you mean now, the methods that execute the request return the error, but if you want to reuse the request instead of executing it, you can't know that value until it is executed IMHO is a legit demand to use a getter for Error(), the field is there but not accessible , WDYT?
/sig api-machinery
a Request#Error() getter seems reasonable
/triage accepted


Question: return nil } <del> func updateDaemonSet(namespace, name string, dynamicClient dynamic.Interface) error { dsClient := dynamicClient.Resource(schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "daemonsets"}).Namespace(namespace) obj, err := dsClient.Get(name, metav1.GetOptions{}) if err != nil { return err } ds := &appsv1.DaemonSet{} if err := runtime.DefaultUnstructuredConverter.FromUnstructured(obj.Object, ds); err != nil { return err } // We set the nodeSelector to a random label. This label is nearly guaranteed // to not be set on any node so the DameonSetController will start deleting // daemon pods. Once it's done deleting the daemon pods, it's safe to delete // the DaemonSet. ds.Spec.Template.Spec.NodeSelector = map[string]string{ string(uuid.NewUUID()): string(uuid.NewUUID()), } // force update to avoid version conflict ds.ResourceVersion = "" out, err := runtime.DefaultUnstructuredConverter.ToUnstructured(ds) if err != nil { return err } if _, err = dsClient.Update(&unstructured.Unstructured{Object: out}); err != nil { return err } // Wait for the daemon set controller to kill all the daemon pods. if err := wait.Poll(1*time.Second, 5*time.Minute, func() (bool, error) { updatedObj, err := dsClient.Get(name, metav1.GetOptions{}) if err != nil { return false, nil } updatedDS := &appsv1.DaemonSet{} if err := runtime.DefaultUnstructuredConverter.FromUnstructured(updatedObj.Object, ds); err != nil { return false, nil } return updatedDS.Status.CurrentNumberScheduled+updatedDS.Status.NumberMisscheduled == 0, nil }); err != nil { return err } return nil } </del> // deletion printing is special because we do not have an object to print. // This mirrors name printer behavior func (o *DeleteOptions) PrintObj(info *resource.Info) {

/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed.

: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Should we create tests around the creation, modification and deletion of the objects ? For example, adding an invalid FlowSchema with a misconfigured PriorityLevelConfiguration and verifying that the Status appropriately reflects its invalidity. I’m assuming the controller tests will suffice and we don’t need explicit e2e tests for this. What about CRUD tests wherein we’re validating the debug endpoints are showing the right values ? The first class of tests are the basic ones to verify that the requests are being categorized into the right FlowSchema and Priority level. We’ll set up a new PriorityLevelConfiguration and FlowSchema objects and observe that requests are correctly being classified onto them. Verification that requests are correctly being classified will be tricky. For now, we can scrape the metrics endpoint for ‘’ metric to verify that the expected number of requests for a test FlowSchema is correct. Another option is to utilize the ‘dump_requests’ debug endpoint to verify the correct request classification. If during the course of writing the tests we need additional information, we can extend the debug endpoint. Scraping the metrics endpoints to validate the proper request categorization could lead to a fickle e2e test. So querying the ‘debug’ endpoint seems like the right way to go. This test will also serve as a proxy for an end-to-end test that ensures that requests matching a new FlowSchema are successful. Does this have the propensity to be a flaky test(s) ? This class of tests will focus on validating the behavior this feature promises, viz., requests of low priority won’t be drowned out by requests of higher priority and a single bad actor won’t detrimentally affect the requests from others. The following tests will ensure that high velocity flows don’t affect the low velocity flows. For this test, we’ll have two entities submitting requests at the same time. These requests will map to two different FlowSchemas at different priority levels. The higher priority flow will have a lot of requests going through it, far above its concurrency share. At the same time the low priority flow will be submitting requests as well at a rate that well within the concurrency limits of that Flow. The test will ensure that the low velocity (low priority) flow succeeds. This test will be the same as the above one except that the two flows will map to the same FlowSchema but are separated via their Flow Distinguisher.

One of them will be a high velocity flow which the other is a low velocity one. The test will ensure that the low velocity requests still get through and succeed. Does this have the propensity to be a flaky tests ? If the test fails, will the runner of this test be able to grok where the problem could lie and debug ? Is this an actionable failure ?
cc/
We don't need to test that invalid items are rejected, unit tests are better for that. We should try to exercise the basic features (exempt, queue...) We should not rely on the debug endpoint, since we want this to be a conformance test eventually Metrics scraping is also a little questionable. I think we should exercise fair queueing (make two new PLs at different shares, saturate both, verify the responses follow the share ratio). I think we should exercise the shuffle sharding (make a new PL, saturate it with one/few elephants and many mice, verify the mice get service. (those latter two seem to match your last two.) This would be a pretty high level test that tests things end users will care about, but I think it should still be quite hard to pass without actually running our code.
/sig api-machinery /area test /kind feature
cc
to verify the requests can be correctly classified, how about we make use of the ? we can do a test like: make two PLs, and two clients w/ different identities then make the clients are hitting different FS/PL.
what about using a test aggregated apiserver to serve different PLs, and with different the response latency
No strong opinion on this one. I am not sure if that was a response to the ideas in the "Basic CRUD" section --- there the idea is not about testing validation but rather testing the status setting about referential integrity. The debug endpoints are about what is happening at the moment of the GET. Controlling that tightly enough to be worth a test would be tricky. The only plausible approach in an E2E test would be to register an authorization webhook that synchronizes with the test function. I like the basic idea. For the reasons above, I would not recommend looking at the debug endpoints. I do not understand what is wrong with scraping the metrics. Are they not part of conformance? This feature does add response headers.

Are those part of conformance? I like the basic idea. First, remember that this feature does not control velocity; it controls the product of velocity and execution duration. To get a high signal-to-noise ratio, we would want deterministic and large execution durations. Looking in , you can see that authorization is included in execution. So we could add a controlled large amount of duration in there. I am a little surprised at the question of "If the test fails, will the runner of this test be able to grok where the problem could lie and debug"; why would that be more of a question here than for any other E2E test? Presumably the test failure message will say something about the expected and observed behavior. I think the drown-out scenario involves testing queuing. I think exempt was not covered above, and would be a worthwhile addition. This mixes up two things. Fair queuing is something that happens within one priority level. This sounds like it may be expensive for an E2E test. I think we can make a much more efficient unit test of QueueSet that exercises shuffle sharding. Because the QueueSet implementation can be given a FakeEventClock, we can run the test much faster than real time. To get a high signal-to-noise ratio we want controlled, high latencies. Making an aggregated apiserver do that is possible. Perhaps the simplest approach would be to put a fixed sleep in the registry strategies for whatever custom resources the aggregated apiserver is serving. But this seems like a lot more work than necessary. I think an authorization webhook is a simpler approach.
could you elaborate a bit more about the authorization webhook approach? it is only called in the authorization path, if i understand it correctly.
Authorization, and admission control, happen after the priority and fairness filter.
is this really possible? I am likely missing something, but my understanding from the is that each level is limited to the concurrency shares assigned to it.
: the text should be revised to more clearly express the intent. It is that the driving client has many more requests outstanding at once than the corresponding priority level will allow to be served concurrently.
FYI: I'm working on the second test.


Question: import ( "flag" "math/rand" <ins> "os" </ins> "os/exec" "time"

When using or with a key/value that has a multi-line string as value, that value will appear as quoted string in the log message, i.e. the log message will still be a single line. It would be nicer to print the value in text mode so that it goes across multiple lines, with clear indicators for start and end of the value. Indention could be used to avoid mistaking some embedded line as the start of a new log message. In JSON mode, the value should be formatted normally. /sig instrumentation /wg structured-logging Always printing as a single line can can make the log output unreadable ().
When adding something like this, can be reverted and implemented differently.
My original thinking was that only objects that explicitly ask for multi-line output get formatted that way. But that means that we need to review log calls very carefully. An alternative is to automatically switch from quoting strings to multi-line output based on some heuristic, for example when there's more than one line break.
Here's a potential implementation:
If structured logging is needed in any other case I can contribute, please inform if needed.
/triage accepted /assign /priority important-soon /area logging /cc /cc
We have the same problem as in kube-scheduler also in kubelet, it's just that no-one has seen and/or complained about the multi-line strings. With the benchmark + log analysis tool from I found the following multi-line string parameters in a kubelet log: <details</details


Question: if err == nil { t.Fatalf("expected the server to send an error, but got nil") } <del> uerr, ok := err.(*url.Error) if !ok { t.Fatalf("expected the error to be of type *url.Error, but got: %T", err) } if !strings.Contains(uerr.Error(), "INTERNAL_ERROR") { t.Fatalf("expected a stream reset error, but got: %s", uerr.Error()) </del> <ins> if isStreamReset(err) { return </ins> } <ins> t.Fatalf("expected a stream reset error, but got %#+v=%s", err, err.Error()) </ins> } func newClientset(t *testing.T, objects ...runtime.Object) clientset.Interface {

pull-kubernetes-unit TestPriorityAndFairnessWithPanicRecoveryAndTimeoutFilter 7/14 No response /sig api-machinery
/priority important-soon
/cc /triage accepted
and can this be related to ?
I see no direct relationship, but hey, anything's possible. We learned that with the misery in integration test failures when we merged not-really-guilty other stuff.
The expanded logging turned up some new evidence. See , where the complaint is now
The problem here is not APF, it is something in the timeout filter or the test server.
is something in APF triggering a timeout?
The flake definitely came in with on the prior commit on master: on the merge commit of
/assign
Note that the flaking test is deliberately suffering server-side timeouts. The symptom in flakes is that the test is checking that the client gets kube's designed response to a timeout while handling, a particular form HTTP response message, but sometimes the client gets something else (an http2 internal error?). is a change in metrics, and should have nothing to do with how timeouts while handling and detected or handled. I will see what I can see...
, regarding your comment above (): note that the output shown (the panic "as designed") is normal; if there are failures involved, the failing part was elided.
Also, the failures seem to be coming from one particular subtest () and that one does not involve designed panics.
On closer examination of the timeout filter, it appears that this "INTERNAL ERROR" is something that can happen normally in the course of handling a timeout. So I will adjust the test to accept this.
I have updated to include the fix for this flake.


Question: }) err = clientretry.RetryOnConflict(UpdateNodeSpecBackoff, func() error { <del> var curNode *v1.Node if cnc.cloud.ProviderName() == "gce" { // TODO(wlan0): Move this logic to the route controller using the node taint instead of condition // Since there are node taints, do we still need this? // This condition marks the node as unusable until routes are initialized in the cloud provider if err := nodeutil.SetNodeCondition(cnc.kubeClient, types.NodeName(nodeName), v1.NodeCondition{ Type:               v1.NodeNetworkUnavailable, Status:             v1.ConditionTrue, Reason:             "NoRouteCreated", Message:            "Node created without a route", LastTransitionTime: metav1.Now(), }); err != nil { return err } // fetch latest node from API server since GCE-specific condition was set and informer cache may be stale curNode, err = cnc.kubeClient.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) if err != nil { return err } } else { curNode, err = cnc.nodeInformer.Lister().Get(nodeName) if err != nil { return err } </del> <ins> curNode, err = cnc.nodeInformer.Lister().Get(nodeName) if err != nil { return err </ins> } newNode := curNode.DeepCopy()

This pertains more for clusters running in GCP but not necessarily in a GKE environment. Clusters that using a CNI supporting overlay such as Calico, do not need explicit routes to be created on the infrastructure level, meaning that the route controller should not be necessary. The following snippet sets the condition for new GCE nodes when a new node is picked up by the node controller. This condition is supposed to be removed later by the route controller in However, as stated above there are cases where the route controller is not needed and should not be running. When Calico is deployed in such a way that the calico-node pods are ignoring any and all taints including the cloudprovider's taint (which we consider reasonable for a system pod such as calico), the above check creates the following race condition: a Calico pod may start and set the node network conditions ( to false) before the node controller reconciles the node. After that the node controller picks up the node and sets the condition to an otherwise healthy node. Leaving aside the race condition itself, I have to ask whether the check is meaningful in the first place. AFAIK, the kubelet in Kubelet only sets the condition for provider, but clusters should already be using kubelet with cloudprovider= For the node controller to not set the condition unnecessarily. An easy way to do it with a custom CCM image is to introduce some delay in the node sync loop, enough for pods such as calico-node to start on that node. No response <details</details<detailsGCP </details<details</details<details</details<details</details<detailsCalico </details
This should be fixed by /cc
/sig cloud-provider /sig network /assign
/triage accepted


Question: // Users should call fakeRuntime.Stop() to cleanup the server. func createAndStartFakeRemoteRuntime(t *testing.T) (*fakeremote.RemoteRuntime, string) { endpoint, err := fakeremote.GenerateEndpoint() <del> assert.NoError(t, err) </del> <ins> require.NoError(t, err) </ins> fakeRuntime := fakeremote.NewFakeRemoteRuntime() <del> go fakeRuntime.Start(endpoint) </del> <ins> fakeRuntime.Start(endpoint) </ins> return fakeRuntime, endpoint } func createRemoteRuntimeService(endpoint string, t *testing.T) internalapi.RuntimeService { runtimeService, err := NewRemoteRuntimeService(endpoint, defaultConnectionTimeout) <del> assert.NoError(t, err) </del> <ins> require.NoError(t, err) </ins> return runtimeService }

A race condition between fakeremote.Start() is spawned as a goroutine to listen for connections, and its caller assumes that the listener will immediately be ready to accept connections. This leads to intermittent test failures if the caller attempts before the fake remote runtime is ready to listen. The code needs to be refactored so that the fake remote returns from listen, and then spawns the server as a goroutine and returns. This can be reproduced by running the remoteruntimetest in a loop; it fails periodically. Failure can be accelerated by putting a time.Sleep() in fake.Start() immediately before the call to util.CreateListener; the failure is then effectively deterministic. Fix tested by running about 150,000 iterations without observing any failures.
/sig pod
Reference initial issue:
/sig node


Question: for i := uint32(0); i < numActual; i++ { ret.Cpu.Usage.PerCpu[i] = s.CpuStats.CpuUsage.PercpuUsage[i] <del> ret.Cpu.Usage.Total += s.CpuStats.CpuUsage.PercpuUsage[i] </del> } }

/kind feature This is the once-per-release cut of a new cAdvisor release, which corresponds to the next kubernetes release. cAdvisor v0.29 corresponds to kubernetes v1.10. /sig node /priority important-soon
self-reminder to add per-cpu stats to ignore metrics when vendoring. reminder to add a release notice that one of the container labels fields is being removed.
/kind bug turns out most of the changes are bug-fixes
please mark this issue as approved-for-milestone, since it certainly seems to be, and it's now code slush. Thanks!


Question: }) // By }) // It f.It("should reject recursive read-only mounts", func(ctx context.Context) { <ins> e2eskipper.SkipUnlessFeatureGateEnabled(features.RecursiveReadOnlyMounts) </ins> ginkgo.By("waiting for the node to be ready", func() { waitForNodeReady(ctx) if supportsRRO(ctx, f) {

<img width="1064" alt="image" src=""### Failure cluster /kind failing-test


Question: ec2Device := "/dev/xvd" + string(mountDevice) if !alreadyAttached { <ins> available, err := c.checkIfAvailable(disk, "attaching", awsInstance.awsID) if !available { attachEnded = true return "", err } </ins> request := &ec2.AttachVolumeInput{ Device:     aws.String(ec2Device), InstanceId: aws.String(awsInstance.awsID),

On AWS, mutable API calls such as AttachVolume or DeleteVolume have separate API quota from API calls that do not modify the resource. Currently k8s makes a high number of AttachVolume and DeleteVolume API calls even though k8s can check if the operation is going to succeed.
/sig aws


Question: Strategy.PrepareForUpdate(ctx, &tc.updatedJob, &tc.job) if diff := cmp.Diff(tc.wantJob, tc.updatedJob); diff != "" { <del> t.Errorf("Job pod failure policy (-want,+got):n%s", diff) </del> <ins> t.Errorf("Job update differences (-want,+got):n%s", diff) </ins> } }) } } <del> // TestJobStrategy_PrepareForUpdate tests various scenearios for PrepareForCreate </del> <ins> // TestJobStrategy_PrepareForCreate tests various scenarios for PrepareForCreate </ins> func TestJobStrategy_PrepareForCreate(t *testing.T) { validSelector := getValidLabelSelector() validPodTemplateSpec := getValidPodTemplateSpecForSelector(validSelector)

The gets very long testing mulitple functions in such as , and . These checks should be performed in dedicated Test functions using table for different scenarios. Target functions and are being as part of The remaining checks should be migrated to these functions. Also, a test function should be to migrate the remaining tests.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Any progress on this? I'm working on and I noticed that has relatively low coverage so I was going to work on getting the coverage up.
I created that is somewhat related to this.
no progess. go ahead please
/assign


Question: Logf(reason) return false, nil } <del> return true, nil </del> <ins> // When the deployment status and its underlying resources reach the desired state, we're done if deployment.Status.Replicas == deployment.Spec.Replicas && deployment.Status.UpdatedReplicas == deployment.Spec.Replicas && deployment.Status.AvailableReplicas == deployment.Spec.Replicas { return true, nil } reason = fmt.Sprintf("deployment status: %#v", deployment.Status) Logf(reason) return false, nil </ins> }) if err == wait.ErrWaitTimeout {

https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Happened on a presubmit run in . Previous issues for this test:
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s-

Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes

e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}


Question: fakeVolumeBinder := volumebinding.NewFakeVolumeBinder(item.volumeBinderConfig) s, bindingChan, errChan := setupTestSchedulerWithVolumeBinding(ctx, fakeVolumeBinder, eventBroadcaster) eventChan := make(chan struct{}) <del> stopFunc := eventBroadcaster.StartEventWatcher(func(obj runtime.Object) { </del> <ins> stopFunc, err := eventBroadcaster.StartEventWatcher(func(obj runtime.Object) { </ins> e, _ := obj.(*eventsv1.Event) if e, a := item.eventReason, e.Reason; e != a { t.Errorf("expected %v, got %v", e, a) } close(eventChan) }) <ins> if err != nil { t.Fatal(err) } </ins> s.scheduleOne(ctx) // Wait for pod to succeed or fail scheduling select {

Follow-up cleanup work for: Most of the information about this issue can be found in , where the work merged with a to change the function signature of to return a func() as well as an error. This issue will help to track that. All TODOs introduced in are resolved. N/A N/A <detailsN/A </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority low Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /triage accepted
/priority important-longterm
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: validate the pod resources are applied to the Limitrange release: v1.18 file: test/e2e/scheduling/limit_range.go <ins> - testname: LimitRange, list, patch and delete a LimitRange by collection codename: '[sig-scheduling] LimitRange should list, patch and delete a LimitRange by collection [Conformance]' description: When two limitRanges are created in different namespaces, both MUST succeed. Listing limitRanges across all namespaces with a labelSelector MUST find both limitRanges. When patching the first limitRange it MUST succeed and the fields MUST equal the new values. When deleting the limitRange by collection with a labelSelector it MUST delete only one limitRange. release: v1.26 file: test/e2e/scheduling/limit_range.go </ins> - testname: Scheduler, resource limits codename: '[sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run  [Conformance]'

4 Oct 2022 [x] Two weeks soak end date: 18 Oct 2022 [x] Test promotion PR: According to this APIsnoop query, there are still three remaining LimitRange endpoints which are untested. listCoreV1LimitRangeForAllNamespacespatchCoreV1NamespacedLimitRangedeleteCoreV1CollectionNamespacedLimitRange Using a number of existing e2e test practices a new has been created for three LimitRange endpoints. The e2e logs for this test are listed below. This query shows which LimitRange endpoints are hit within a short period of running this e2e test example endpoint | useragent --------------------------------------------+
createCoreV1NamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection deleteCoreV1CollectionNamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection listCoreV1LimitRangeForAllNamespaces | should ensure that a limitRange can be listed, patched and deleted by collection listCoreV1NamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection patchCoreV1NamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection (5 rows) If a test with these calls gets merged, test coverage will go up by 3 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // TLSConfigFor returns a tls.Config that will provide the transport level security defined // by the provided Config. Will return nil if no transport level security is requested. func TLSConfigFor(config *Config) (*tls.Config, error) { <del> cfg, err := config.transportConfig() </del> <ins> cfg, err := config.TransportConfig() </ins> if err != nil { return nil, err }

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}


Question: --network ${NETWORK}  --region ${REGION}  --range ${NODE_IP_RANGE}  <del> --secondary-range "pods-default=${CLUSTER_IP_RANGE}" </del> <ins> --secondary-range "pods-default=${CLUSTER_IP_RANGE}"  --secondary-range "services-default=${SERVICE_CLUSTER_IP_RANGE}" </ins> echo "Created subnetwork ${IP_ALIAS_SUBNETWORK}" else if ! echo ${subnet} | grep --quiet secondaryIpRanges ${subnet}; then

When using IP aliases on GCE, use a secondary range rather than subnetwork to reserve cluster IPs.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


Question: func setSandboxResources(hc *dockercontainer.HostConfig) { hc.Resources = dockercontainer.Resources{ <del> MemorySwap: -1, </del> <ins> MemorySwap: dockertools.DefaultMemorySwap(), </ins> CPUShares:  defaultSandboxCPUshares, // Use docker's default cpu quota/period. }

What happened: Latest kubelet + Windows Docker version 1.14.0-dev refused to start any containers, error output: "invalid option: Windows does not support MemorySwap" How to reproduce it (as minimally and precisely as possible): It's always reproducible with Windows Docker 1.14.0-dev + latest kubelet code base. Anything else do we need to know: It didn't happen for Windows Docker 1.12.2-cs2-ws-beta. The reason behind the issue was because new code was for Windows Docker to check Resources parameters, it doesn't accept a non-zero value for MemorySwap: But kubelet was default to set an unlimited (-1) value. The issue was gone in my setup after applied a private fix to disable MemorySwap by set it to zero for Windows.


Question: // (e.g. master nodes) may not belong to any scale sets. availabilitySet VMSet <ins> vmssCache                 *timedCache </ins> vmssVMCache               *timedCache availabilitySetNodesCache *timedCache }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Recently we upgraded our cluster from 1.14.4 to 1.14.7. Then we started to observe tons of following errors: I guess the cause may be related to this line which was introduced recently: As we have hundreds of services with LoadBalancer type, and we also enable cluster-autoscaler, the nodes list change very often. For each service, there will be one call to GetScaleSet. The quota will be exhausted quickly. When fails, new nodes created by cluster-autoscaler will lose connectivity to load balancer. What you expected to happen: Kubernetes should not hit VMSS throttling threshold. How to reproduce it (as minimally and precisely as possible): a cluster with aks-engine. Use VMSS. 200 services with LoadBalancer type. up VMSS to 50 nodes down VMSS to 60 nodes scaling up and down VMSS Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.7 Cloud provider or hardware configuration: Azure, VMSS OS (e.g: ): Ubuntu Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure
I create a PR to mitigate the issue.
do we have a rough num about how this PR would mitigate this issue? looks like we need to cherry-pick this PR to old releases. it's better if we have a rough estimation.
The revised version of code calls instead of azure API. The cache refreshes itself every 10 min. I believe this will solve the throttling issue.
I believe we're being affected by this. Apart from getting the - when creating our to use a pre-configured Azure Public IP, the ingress appears to be getting a different IP each time
Thank you all!
The PR is merged. Does your issue remain?

pls cherry-pick that fix to old release, thanks
Cherry picked to release 1.14-1.16
I’ll retry today and let you know Wysłane z iPhone'a
Issue returned in K8S 1.15.4 Warning UpdateLoadBalancerFailed service/nginx-ingress-controller Error updating load balancer with new hosts map[<nodes names]: [EnsureHostInPool(ingress/nginx-ingress-controller): backendPoolID(/subscriptions/<subscription/resourceGroups/<rg/providers/Microsoft.Network/loadBalancers/<rg/backendAddressPools/<rg) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:VMSSGet", EnsureHostInPool(ingress/nginx-ingress-controller):


Question: } EOF if [[ "${KUBERNETES_MASTER:-}" != "true" ]]; then <del> if [[ "${NETWORK_POLICY_PROVIDER:-"none"}" != "none" || "${ENABLE_NETD:-}" == "true" ]]; then # Use Kubernetes cni daemonset on node if network policy provider is specified </del> <ins> if [[ "${NETWORK_POLICY_PROVIDER:-"none"}" == "calico" || "${ENABLE_NETD:-}" == "true" ]]; then # Use Kubernetes cni daemonset on node if network policy provider calico is specified </ins> # or netd is enabled. cni_template_path="" fi

,%20google-gce all I can't say , but at least a few weeks ,%20google-gce No response No response /sig network
/assign
/triage accepted


Question: "+someVersionTag:version=v1.7", }, wantValue: nil, <del> wantErr:   true, // TODO: Today it is klog.Fatal, check how to capture it </del> <ins> wantFatal: true, </ins> }, { name:    "multiple values on same tag should return an error",

The commands under will benefit of a batter unit test coverage as seen in Originally posted by in
/sig api-machinery /assign
GitHub didn't allow me to assign the following users: damiansawicki. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Hi I would like to work on this issue. Could you tell me how many test files are needed per PR under this issue?
Hi the issue is already assigned to me. Would you consider contributing to other issues to avoid duplication?
Hi do you mind if I work on this issue along with you? I would only like to write tests on limited number of files, if that's okay with you...
Hi thanks for your engagement! If you also want to work on this particular issue, we should coordinate. Are you working on something other than ?
Hey currently I'm thinking of shifting to Do let me know if you are already working on that particular directory. Also I could assist you in any of your work on this issue as I'm not preoccupied at the moment.


Question: // StartStructuredLogging starts sending events received from this EventBroadcaster to the structured logging function. // The return value can be ignored or used to stop recording, if desired. <ins> // TODO: this function should also return an error. </ins> func (e *eventBroadcasterImpl) StartStructuredLogging(verbosity klog.Level) func() { <del> return e.StartEventWatcher( </del> <ins> stopWatcher, err := e.StartEventWatcher( </ins> func(obj runtime.Object) { event, ok := obj.(*eventsv1.Event) if !ok {

Follow-up cleanup work for: Most of the information about this issue can be found in , where the work merged with a to change the function signature of to return a func() as well as an error. This issue will help to track that. All TODOs introduced in are resolved. N/A N/A <detailsN/A </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority low Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /triage accepted
/priority important-longterm
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: http_file( name = "kubernetes_cni", <del> sha256 = "ddcb7a429f82b284a13bdb36313eeffd997753b6fa5191205f1e978dcfeb0792", url = " https://storage.googleapis.com/kubernetes-release/network-plugins/cni-amd64-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz", </del> <ins> sha256 = "05ab3937bc68562e989dc143362ec4d4275262ba9f359338aed720fc914457a5", url = "https://storage.googleapis.com/kubernetes-release/network-plugins/cni-amd64-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz", </ins> )

Started as a tangent of Many CNI plugins don't properly handle a action on missing resources. The spec says, , however many improperly return an error. I see the steps moving forward to be: [x] Modify CNI plugins bundled with Kubernetes to properly support del (flannel already done; bridge, ipvlan, macvlan, and ptp in containernetworking/cni) [x] Create release of containernetworking/cni Does this make sense? cc/
SGTM - I'm working on the right now and can do the CNI release too. On the 3rd check box - why is that still a thing? Doesn't the CNI project provide the required artifacts as part of its release? Maybe not something to resolve now but I'd like to make sure the CNI project is providing the right things to projects that use it.
I just want to say thanks to everyone for jumping on this.
status tracking doc
I can test out patches as needed. I currently only have test cases for Weave and Calico, but it should not be too difficult to add others.
It would be great to get flannel tested too. You probably need a master build of the flannel CNI plugin and possibly some flannel patches to be merged. Feel free to DM on slack if you need more info.
s390x builds are missing still. If that's , we can switch to v0.5.1 in time for v1.7. However, I really don't want to bump from v0.3.0-ish to v0.5.0 one day before release. This is a v1.7 issue to ensure all CNI plugins (including the official) work as intended with Kubernetes and an issue to make sure we update to CNI v0.5.0 w/ spec v0.3.0 in time for v1.7. I think will solve the short-term solutions. Also it seems like Weave, Calico and maybe Canal has fixed this as-is. I think Flannel works with
I'm a little confused! You mean s390x builds for flanneld? And what do you mean by This is a v1.7 issue I thought this was a 1.6 issue?

CNI still needs s390x binaries to be consumable by k8s: Two other conclusions: 1) We should go with as the immediate solution 2) We should upgrade to CNI v0.5.1 (when released( officially for Kubernetes in time for v1.7 and enforce the latest spec (v0.3.0) for all network plugins
We should get the PR in (after testing), but this is not a really solution as it only avoids hitting this problem at cluster setup.
OK, that makes sense on the s390x binaries. I've just released v0.5.1 of CNI, which updates the plugins with the new del behaviour. They should all still be back compatible with the older spec versions so I recommend that K8s takes this CNI release.
I'm working on a patch for that bump now
Be aware that has just been merged. If you sync to HEAD, you won't be able to reproduce this problem at cluster setup easily.
New cni packaged tarball (based on v0.5.1) has been pushed:
-- can this be closed?
we can't close until all the consumers have been updated to point to the new tarballs
What is the ETA for resolving this?
Ok, I see that is at the top of the submit queue.


Question: } } } <ins> // TestAllocatableNodeFsPressure func TestAllocatableNodeFsPressure(t *testing.T) { podMaker := makePodWithDiskStats summaryStatsMaker := makeDiskStats podsToMake := []podToMake{ {name: "guaranteed-low", requests: newEphemeralStorageResourceList("200Mi", "100m", "1Gi"), limits: newEphemeralStorageResourceList("200Mi", "100m", "1Gi"), rootFsUsed: "200Mi"}, {name: "guaranteed-high", requests: newEphemeralStorageResourceList("800Mi", "100m", "1Gi"), limits: newEphemeralStorageResourceList("800Mi", "100m", "1Gi"), rootFsUsed: "800Mi"}, {name: "burstable-low", requests: newEphemeralStorageResourceList("300Mi", "100m", "100Mi"), limits: newEphemeralStorageResourceList("300Mi", "200m", "1Gi"), logsFsUsed: "300Mi"}, {name: "burstable-high", requests: newEphemeralStorageResourceList("800Mi", "100m", "100Mi"), limits: newEphemeralStorageResourceList("800Mi", "200m", "1Gi"), rootFsUsed: "800Mi"}, {name: "best-effort-low", requests: newEphemeralStorageResourceList("300Mi", "", ""), limits: newEphemeralStorageResourceList("300Mi", "", ""), logsFsUsed: "300Mi"}, {name: "best-effort-high", requests: newEphemeralStorageResourceList("800Mi", "", ""), limits: newEphemeralStorageResourceList("800Mi", "", ""), rootFsUsed: "800Mi"}, } pods := []*v1.Pod{} podStats := map[*v1.Pod]statsapi.PodStats{} for _, podToMake := range podsToMake { pod, podStat := podMaker(podToMake.name, podToMake.requests, podToMake.limits, podToMake.rootFsUsed, podToMake.logsFsUsed, podToMake.perLocalVolumeUsed) pods = append(pods, pod) podStats[pod] = podStat } podToEvict := pods[5] activePodsFunc := func() []*v1.Pod { return pods } fakeClock := clock.NewFakeClock(time.Now()) podKiller := &mockPodKiller{} diskInfoProvider := &mockDiskInfoProvider{dedicatedImageFs: false} capacityProvider := newMockCapacityProvider(newEphemeralStorageResourceList("6Gi", "1000m", "10Gi"), newEphemeralStorageResourceList("1Gi", "1000m", "10Gi")) diskGC := &mockDiskGC{imageBytesFreed: int64(0), err: nil} nodeRef := &v1.ObjectReference{Kind: "Node", Name: "test", UID: types.UID("test"), Namespace: ""} config := Config{ MaxPodGracePeriodSeconds: 5, PressureTransitionPeriod: time.Minute * 5, Thresholds: []evictionapi.Threshold{ { Signal:   evictionapi.SignalAllocatableNodeFsAvailable, Operator: evictionapi.OpLessThan, Value: evictionapi.ThresholdValue{ Quantity: quantityMustParse("1Ki"), }, }, }, } summaryProvider := &fakeSummaryProvider{result: summaryStatsMaker("3Gi", "6Gi", podStats)} manager := &managerImpl{ clock:           fakeClock, killPodFunc:     podKiller.killPodNow, imageGC:         diskGC, containerGC:     diskGC, config:          config, recorder:        &record.FakeRecorder{}, summaryProvider: summaryProvider, nodeRef:         nodeRef, nodeConditionsLastObservedAt: nodeConditionsObservedAt{}, thresholdsFirstObservedAt:    thresholdsObservedAt{}, } // create a best effort pod to test admission bestEffortPodToAdmit, _ := podMaker("best-admit", newEphemeralStorageResourceList("", "", ""), newEphemeralStorageResourceList("", "", ""), "0Gi", "", "") burstablePodToAdmit, _ := podMaker("burst-admit", newEphemeralStorageResourceList("1Gi", "", ""), newEphemeralStorageResourceList("1Gi", "", ""), "1Gi", "", "") // synchronize manager.synchronize(diskInfoProvider, activePodsFunc, capacityProvider) // we should not have disk pressure if manager.IsUnderDiskPressure() { t.Fatalf("Manager should not report disk pressure") } // try to admit our pods (they should succeed) expected := []bool{true, true} for i, pod := range []*v1.Pod{bestEffortPodToAdmit, burstablePodToAdmit} { if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: pod}); expected[i] != result.Admit { t.Fatalf("Admit pod: %v, expected: %v, actual: %v", pod, expected[i], result.Admit) } } // induce disk pressure! fakeClock.Step(1 * time.Minute) pod, podStat := podMaker("guaranteed-high-2", newEphemeralStorageResourceList("2000Mi", "100m", "1Gi"), newEphemeralStorageResourceList("2000Mi", "100m", "1Gi"), "2000Mi", "", "") podStats[pod] = podStat pods = append(pods, pod) summaryProvider.result = summaryStatsMaker("6Gi", "6Gi", podStats) manager.synchronize(diskInfoProvider, activePodsFunc, capacityProvider) // we should have disk pressure if !manager.IsUnderDiskPressure() { t.Fatalf("Manager should report disk pressure") } // check the right pod was killed if podKiller.pod != podToEvict { t.Fatalf("Manager chose to kill pod: %v, but should have chosen %v", podKiller.pod, podToEvict.Name) } observedGracePeriod := *podKiller.gracePeriodOverride if observedGracePeriod != int64(0) { t.Fatalf("Manager chose to kill pod with incorrect grace period.  Expected: %d, actual: %d", 0, observedGracePeriod) } // reset state podKiller.pod = nil podKiller.gracePeriodOverride = nil // try to admit our pod (should fail) expected = []bool{false, false} for i, pod := range []*v1.Pod{bestEffortPodToAdmit, burstablePodToAdmit} { if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: pod}); expected[i] != result.Admit { t.Fatalf("Admit pod: %v, expected: %v, actual: %v", pod, expected[i], result.Admit) } } // reduce disk pressure fakeClock.Step(1 * time.Minute) pods[5] = pods[len(pods)-1] pods = pods[:len(pods)-1] // we should have disk pressure (because transition period not yet met) if !manager.IsUnderDiskPressure() { t.Fatalf("Manager should report disk pressure") } // try to admit our pod (should fail) expected = []bool{false, false} for i, pod := range []*v1.Pod{bestEffortPodToAdmit, burstablePodToAdmit} { if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: pod}); expected[i] != result.Admit { t.Fatalf("Admit pod: %v, expected: %v, actual: %v", pod, expected[i], result.Admit) } } // move the clock past transition period to ensure that we stop reporting pressure fakeClock.Step(5 * time.Minute) manager.synchronize(diskInfoProvider, activePodsFunc, capacityProvider) // we should not have disk pressure (because transition period met) if manager.IsUnderDiskPressure() { t.Fatalf("Manager should not report disk pressure") } // no pod should have been killed if podKiller.pod != nil { t.Fatalf("Manager chose to kill pod: %v when no pod should have been killed", podKiller.pod.Name) } // all pods should admit now expected = []bool{true, true} for i, pod := range []*v1.Pod{bestEffortPodToAdmit, burstablePodToAdmit} { if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: pod}); expected[i] != result.Admit { t.Fatalf("Admit pod: %v, expected: %v, actual: %v", pod, expected[i], result.Admit) } } } func TestNodeReclaimForAllocatableFuncs(t *testing.T) { podMaker := makePodWithDiskStats summaryStatsMaker := makeDiskStats podsToMake := []podToMake{ {name: "guaranteed-low", requests: newEphemeralStorageResourceList("200Mi", "100m", "1Gi"), limits: newEphemeralStorageResourceList("200Mi", "100m", "1Gi"), rootFsUsed: "200Mi"}, {name: "guaranteed-high", requests: newEphemeralStorageResourceList("800Mi", "100m", "1Gi"), limits: newEphemeralStorageResourceList("800Mi", "100m", "1Gi"), rootFsUsed: "800Mi"}, {name: "burstable-low", requests: newEphemeralStorageResourceList("300Mi", "100m", "100Mi"), limits: newEphemeralStorageResourceList("300Mi", "200m", "1Gi"), logsFsUsed: "300Mi"}, {name: "burstable-high", requests: newEphemeralStorageResourceList("800Mi", "100m", "100Mi"), limits: newEphemeralStorageResourceList("800Mi", "200m", "1Gi"), rootFsUsed: "800Mi"}, {name: "best-effort-low", requests: newEphemeralStorageResourceList("300Mi", "", ""), limits: newEphemeralStorageResourceList("300Mi", "", ""), logsFsUsed: "300Mi"}, {name: "best-effort-high", requests: newEphemeralStorageResourceList("800Mi", "", ""), limits: newEphemeralStorageResourceList("800Mi", "", ""), rootFsUsed: "800Mi"}, } pods := []*v1.Pod{} podStats := map[*v1.Pod]statsapi.PodStats{} for _, podToMake := range podsToMake { pod, podStat := podMaker(podToMake.name, podToMake.requests, podToMake.limits, podToMake.rootFsUsed, podToMake.logsFsUsed, podToMake.perLocalVolumeUsed) pods = append(pods, pod) podStats[pod] = podStat } podToEvict := pods[5] activePodsFunc := func() []*v1.Pod { return pods } fakeClock := clock.NewFakeClock(time.Now()) podKiller := &mockPodKiller{} diskInfoProvider := &mockDiskInfoProvider{dedicatedImageFs: false} capacityProvider := newMockCapacityProvider(newEphemeralStorageResourceList("6Gi", "1000m", "10Gi"), newEphemeralStorageResourceList("1Gi", "1000m", "10Gi")) imageGcFree := resource.MustParse("800Mi") diskGC := &mockDiskGC{imageBytesFreed: imageGcFree.Value(), err: nil} nodeRef := &v1.ObjectReference{Kind: "Node", Name: "test", UID: types.UID("test"), Namespace: ""} config := Config{ MaxPodGracePeriodSeconds: 5, PressureTransitionPeriod: time.Minute * 5, Thresholds: []evictionapi.Threshold{ { Signal:   evictionapi.SignalAllocatableNodeFsAvailable, Operator: evictionapi.OpLessThan, Value: evictionapi.ThresholdValue{ Quantity: quantityMustParse("10Mi"), }, }, }, } summaryProvider := &fakeSummaryProvider{result: summaryStatsMaker("6Gi", "6Gi", podStats)} manager := &managerImpl{ clock:           fakeClock, killPodFunc:     podKiller.killPodNow, imageGC:         diskGC, containerGC:     diskGC, config:          config, recorder:        &record.FakeRecorder{}, summaryProvider: summaryProvider, nodeRef:         nodeRef, nodeConditionsLastObservedAt: nodeConditionsObservedAt{}, thresholdsFirstObservedAt:    thresholdsObservedAt{}, } // synchronize manager.synchronize(diskInfoProvider, activePodsFunc, capacityProvider) // we should not have disk pressure if manager.IsUnderDiskPressure() { t.Errorf("Manager should not report disk pressure") } // induce hard threshold fakeClock.Step(1 * time.Minute) pod, podStat := podMaker("guaranteed-high-2", newEphemeralStorageResourceList("2000Mi", "100m", "1Gi"), newEphemeralStorageResourceList("2000Mi", "100m", "1Gi"), "2000Mi", "", "") podStats[pod] = podStat pods = append(pods, pod) summaryProvider.result = summaryStatsMaker("6Gi", "6Gi", podStats) manager.synchronize(diskInfoProvider, activePodsFunc, capacityProvider) // we should have disk pressure if !manager.IsUnderDiskPressure() { t.Fatalf("Manager should report disk pressure since soft threshold was met") } // verify image gc was invoked if !diskGC.imageGCInvoked || !diskGC.containerGCInvoked { t.Fatalf("Manager should have invoked image gc") } // verify no pod was killed because image gc was sufficient if podKiller.pod == nil { t.Fatalf("Manager should have killed a pod, but not killed") } // check the right pod was killed if podKiller.pod != podToEvict { t.Fatalf("Manager chose to kill pod: %v, but should have chosen %v", podKiller.pod, podToEvict.Name) } observedGracePeriod := *podKiller.gracePeriodOverride if observedGracePeriod != int64(0) { t.Fatalf("Manager chose to kill pod with incorrect grace period.  Expected: %d, actual: %d", 0, observedGracePeriod) } // reset state diskGC.imageGCInvoked = false diskGC.containerGCInvoked = false podKiller.pod = nil podKiller.gracePeriodOverride = nil // reduce disk pressure fakeClock.Step(1 * time.Minute) pods[5] = pods[len(pods)-1] pods = pods[:len(pods)-1] // we should have disk pressure (because transition period not yet met) if !manager.IsUnderDiskPressure() { t.Fatalf("Manager should report disk pressure") } // move the clock past transition period to ensure that we stop reporting pressure fakeClock.Step(5 * time.Minute) manager.synchronize(diskInfoProvider, activePodsFunc, capacityProvider) // we should not have disk pressure (because transition period met) if manager.IsUnderDiskPressure() { t.Fatalf("Manager should not report disk pressure") } // no pod should have been killed if podKiller.pod != nil { t.Fatalf("Manager chose to kill pod: %v when no pod should have been killed", podKiller.pod.Name) } // no image gc should have occurred if diskGC.imageGCInvoked || diskGC.containerGCInvoked { t.Errorf("Manager chose to perform image gc when it was not neeed") } // no pod should have been killed if podKiller.pod != nil { t.Errorf("Manager chose to kill pod: %v when no pod should have been killed", podKiller.pod.Name) } } </ins>

It is possible to have multiple signals that point to the same type of resource, e.g., both SignalNodeFsAvailable and SignalAllocatableNodeFsAvailable refer to the same resource NodeFs. Change the map from map[v1.ResourceName]evictionapi.Signal to map[v1.ResourceName][]evictionapi.Signal
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/kind cleanup /priority important/soon taking a guess at labels to satisfy the milestone munger
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Looks like the PR for this is just awaiting merge


Question: } if err := r.watchHandler(w, &resourceVersion, resyncCh, stopCh); err != nil { if err != errorResyncRequested && err != errorStopRequested { <del> util.HandleError(fmt.Errorf("%s: watch of %v ended with: %v", r.name, r.expectedType, err)) </del> <ins> glog.Warningf("%s: watch of %v ended with: %v", r.name, r.expectedType, err) </ins> } return }

The same logs for controller and scheduler. According to , it should be a bug. Some insist it might do no harm though. (all containerized) platform: ubuntu 14.04 docker:1.7.1 /cc
You are not experiencing , your watches are lasting many minutes. would you like to downgrade these messages to warnings so we'll stop getting these reports?
Sure.
If that's the case, i shall wait for the updates. And many thanks.


Question: return n.info.Node(), nil } <ins> // updateMetrics updates cache size metric values for pods, assumed pods, and nodes func (cache *schedulerCache) updateMetrics() { metrics.CacheSize.WithLabelValues("assumed_pods").Set(float64(len(cache.assumedPods))) metrics.CacheSize.WithLabelValues("pods").Set(float64(len(cache.podStates))) metrics.CacheSize.WithLabelValues("nodes").Set(float64(len(cache.nodes))) } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a metric to track scheduler cache size. metric name Description -- -- schedulercachesize Number of elements in scheduler cache. Why is this needed: Gives better observability of scheduler cache. For, the number of "assumedpods" gives us an idea of how many pods are "assumed" and therefore being "bound". We can compare this with number of to make sure assume-bind works as expected. /sig scheduling
Is anyone already working on these metrics? If not I'd like to help out /assign
Thanks . Feel free to work on this one. Just a reminder, we may need feedback/approval from sig-instrumentation.


Question: fakeClient := &fake.Clientset{} fakeClient.AddReactor("list", "pods", func(action core.Action) (handled bool, ret runtime.Object, err error) { obj := &v1.PodList{} <del> for i := 0; i < int(tc.currentReplicas); i++ { </del> <ins> podsCount := int(tc.currentReplicas) // Failed pods are not included in tc.currentReplicas if tc.podPhase != nil && len(tc.podPhase) > podsCount { podsCount = len(tc.podPhase) } for i := 0; i < podsCount; i++ { </ins> podReadiness := v1.ConditionTrue <del> if tc.podReadiness != nil { </del> <ins> if tc.podReadiness != nil && i < len(tc.podReadiness) { </ins> podReadiness = tc.podReadiness[i] } <ins> podPhase := v1.PodRunning if tc.podPhase != nil { podPhase = tc.podPhase[i] } </ins> podName := fmt.Sprintf("%s-%d", podNamePrefix, i) pod := v1.Pod{ Status: v1.PodStatus{ <del> Phase: v1.PodRunning, </del> <ins> Phase: podPhase, </ins> Conditions: []v1.PodCondition{ { Type:   v1.PodReady,

/kind bug What happened: HPA fails to launch additional pods due to pods in evicted state, even with utilization significantly over target. What you expected to happen: When HPA pods reach the target utilization, new pods should be allocated and provisioned. How to reproduce it (as minimally and precisely as possible): Launch a HPA, let the pods fill the disk on the host which will trigger evictions due to "Pod The node was low on resource: [DiskPressure]", HPA will exceed target and not allocate additional pods. Anything else we need to know?: Seems related to and though those are not specific to HPA. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): cos-stable-57-9202-64-0 kernel commit id Kernel (e.g. ):Linux gke-prod1-use1b-v2--35s4 4.4.70+ SMP Mon Sep 18 12:43:08 PDT 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux
/sig scheduling
Are there any workarounds while we wait for a fix? We have ran into this on a 1.8.4 GKE cluster.
/sig autoscaling /remove-sig scheduling
I am thinking of fixing this by excluding pods with Phase=Failed from the unreadyPods that are taken into account. It seems the right thing to do as they will not become running again. Do you see any issues?
that seems decent -- anything that will never be running again should not be factored into our calculations.
/status approved-for-milestone Adding to milestone -- if the HPA isn't functioning in the case of evictions, that seems like a decently severe HPA bug.
oops, missing priority was my fault, moving back into milestone: /priority critical-urgent /remove-priority important-soon
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues.

Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details