Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" utilyaml "k8s.io/apimachinery/pkg/util/yaml" clientset "k8s.io/client-go/kubernetes" <ins> cliflag "k8s.io/component-base/cli/flag" </ins> commontest "k8s.io/kubernetes/test/e2e/common" "k8s.io/kubernetes/test/e2e/framework" e2econfig "k8s.io/kubernetes/test/e2e/framework/config"

What happened: During the work for reducing invalid dependencies from the e2e core framework to subpackages of the core framework, we found a lot of particular test specific functions. That means these each function is used at a single e2e test module only. These functions are unnecessary to be shared as e2e framework function, so we've just moved these functions to the particular test module like This cleanup work will help us to maintain the e2e core framework in long term, so it is helpful to share this work in volunteers as a community. This issue is for tracking this working items. How to do that?: volunteer selects a single file which is empty (no PR, no assignee) from the following e2e core framework all functions in the file all callers of each function finding the function is called at a single e2e test package only, let's move the function to the module with a PR Implementation rules: If a function is called from 2+ test packages, don't move the function from the core package. If moving of the function requires invalid code dependencies, don't move the function. (Dependency from e2e core framework to other repository) List of files of e2e core framework: [x] ( ) [x] () [x] ( [x] () [x] [x] () [x] get-kubemark-resource- [x] ( ) [x] (Done) [x] ( ) [x] () [x] ( ) [x] ( ) [x] ( ) [x] ( ) [x] ( ) [x] ( ) [x] () [x] ( ) [x] () [x] () References: This working item was under discussion as
/kind cleanup /remove-kind bug /area e2e-test-framework /sig testing /priority important-longterm
Thanks for taking care of this quickly :-)
The proposed policy to follow for what methods are to be kept within the framework: What do you think ? does that sound good?
Thanks for bring it up here. As I said on the slack, I agree with his idea.
I looked through files , and , and found some of the functions are used in a single module. Then fixed them.

i will pick up
I will pick
I will pick up
/assign I will pick
i will pick and
I will pick up ,,,
The PR for is already created by . Please check what other people picked already before posting a PR.
is already picked/posted by
That is a nice point. I wrote point as one of the above Implementation Rules, and another rule comes from which adds another invalid dependency.
I didn't noticed that already picked by , sorry about that. After looking around, there is noting need to do with: ,,,,(not picked). I will pick the left: . Found that is also no need to move, is the same.
Thanks for your investigation, I've updated the above list based on your input :-)
I think is does not need to be modified, Cloud you please help me confirm it?
and
There is noting need to do with , and I have filed a PR for
Thanks for your help :-) I've updated the above list based on your input.
Thank you also, I approved the PR and updated the above list from your work.
and do not need to be modified. Could you help me confirm it? THX.
I think there is noting need to do with
Thanks for your work, I've updated the above list based on your input.
I checked but there is no func which is called from less than 2 test packages.
Thanks for your investigation, I've updated the above list based on your input.
Nice!
Yes, thank you so much everyone!! :-)


Question: return nil }, }, <ins> { // delete success(?) - volume has deletion timestamp before doDelete() starts "8-13 - volume is has deletion timestamp and is not processed", withVolumeDeletionTimestamp(newVolumeArray("volume8-13", "1Gi", "uid8-13", "claim8-13", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty)), withVolumeDeletionTimestamp(newVolumeArray("volume8-13", "1Gi", "uid8-13", "claim8-13", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty)), noclaims, noclaims, noevents, noerrors, // We don't need to do anything in test function because deletion will be noticed automatically and synced. // Attempting to use testSyncVolume here will cause an error because of race condition between manually // calling testSyncVolume and volume loop running. func(ctrl *PersistentVolumeController, reactor *pvtesting.VolumeReactor, test controllerTest) error { return nil }, }, </ins> } for _, test := range tests {

PV controller appears to be processing same object multiple times for same PVC delete event. The reason is mainly because when a PVC is deleted and PV is synced(either because of an event or because of resync), then each update to PV object causes another event which gets processed until PV object is removed from informer/store and really marked as deleted. This results in multiple delete requests being sent to storage provider for same volume. For example - In above gist you can see that volume deletion is being done more than once for pv - /sig storage cc
It looks like this is to some extent by design because events are pulled from workqueue by one goroutine and then do some proccessing and then put into another goroutine via nestedpendingoperations thingy. The problem is, this is inherently "racy" because first goroutine might be pulling events that were triggered by second goroutine and hence causing processing of events that are still in-progress. I take we can't use workqueues with a fixed number of workers because that would kinda limit number of PVs we can provision concurrently, but can we put the item pulled from queue directly into nested_pendingoperations so as we don't have this weird race condition?


Question: for _, initDriver := range testDrivers { curDriver := initDriver(config) <ins> curConfig := curDriver.GetDriverInfo().Config </ins> Context(testsuites.GetDriverNameWithFeatureTags(curDriver), func() { <del> driver := curDriver </del> BeforeEach(func() { <ins> // Reset config. The driver might have modified its copy // in a previous test. curDriver.GetDriverInfo().Config = curConfig </ins> // setupDriver <del> driver.CreateDriver() </del> <ins> curDriver.CreateDriver() </ins> }) AfterEach(func() { // Cleanup driver <del> driver.CleanupDriver() </del> <ins> curDriver.CleanupDriver() </ins> }) <del> testsuites.RunTestSuite(f, driver, testSuites, intreeTunePattern) </del> <ins> testsuites.RunTestSuite(f, curDriver, testSuites, intreeTunePattern) </ins> }) } })

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce/gke multizone/regional jobs Which test(s) are failing: Only PD tests Since when has it been failing: Around 12/21 with the 16:49 run Testgrid link: https://k8s- Reason for failure: TBD Anything else we need to know: <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
The runs started flaking after merged. cc Here's one failing run: Pod pod-subpath-test-gcepd-dynamicpv-pfsm failed to schedule because of conflicting node selectors: We shouldn't need to set the node selector for PD tests using PVs.
The PR broke the subpath test because the modification to the config struct embedded in the driver info now persists across tests: A quick fix would be to reset that field in CreateDriver. The long-term fix is the change discussed in My preference is to do the quick fix now, then either merge or close PRs in this order: (needs a decision about logging of tests that aren't getting defined, see ) a future PR for What I'd like to avoid is having to rebase PR on top of the long-term solution for issue - that'll be lots of code conflicts.
/assign PR is here: PR


Question: spec: containers: - name: kubedns <del> image: gcr.io/google_containers/kubedns-{{ arch }}:1.6 </del> <ins> image: gcr.io/google_containers/kubedns-{{ arch }}:1.7 </ins> resources: # TODO: Set memory limits when we've profiled the container for large # clusters, then set request = limit to keep this container in

Now that merged, we need to build a new kube-dns release.
I just put in a PR yesterday to fix the missing changelog entry for kube-dns:1.6


Question: // When a delete is dropped, the relist will notice a pod in the store not // in the list, leading to the insertion of a tombstone object which contains <del> // the deleted key/value. Note that this value might be stale. If the pod // changed labels the new StatefulSet will not be woken up till the periodic resync. </del> <ins> // the deleted key/value. Note that this value might be stale. </ins> if !ok { tombstone, ok := obj.(cache.DeletedFinalStateUnknown) if !ok {

<!-- Please only use this template for submitting enhancement requests --/area controller-manager What would you like to be : Don't resync statefulset in statefulset controller. Why is this needed: There are 2000+ sts in my k8s cluster, thus when i create a new sts, it takes a lot of time to create it's pod, i found that every sts will cost 40ms to sync it, then 2000 sts will cost 80s, and there is also a statefulSetResyncPeriod to resync all sts, I don't know why there need to have statefulSetResyncPeriod to resync all sts? Can we not resync sts and just watch it's changes?
/sig api-machinery
/remove-sig api-machinery /sig apps /sig scalability
the resync pattern is followed by all controllers to account for bugs in watch mechanism or missed notification due to any other reason. I dont think the individual controllers resync period is configurable or we could increase it . Also i am not sure if the current scale limits of 2000 StatefulSets is supported FYI
I have three cluster, A has 20 sts, B has 1300 sts and C has 2300 sts. I set kube-controller-manager loglevel to 4 and find that the cost of processing every sts is 2ms,30ms,45ms. Can we record the processed object into a processed cache and every time we get a new object to precess, we can compare it with the item in the cache, if they are same, then we don't need to sync it.
The "resync pattern" doesn't help with watch bugs (resync is done purely from internal cache, so in hypothetical situation that we missed watch event, it won't help at all). Resync-period should basically be used only in controllers that also sync some external things (like load-balancers), and watching changes of those is impossible. Unless I'm not missing anything, resync period can be safely dropped in this case.
I think all the controllers have the same problem because of the sharedinformerfactory has a default resync period if we don't set it to 0 in the config file. Also if there are more sts, the consumption for every sts will be larger.
interesting, but the resync is only happening for k8s objects.

Why is watching changes on those impossible ? And when you say external things you mean loadbalancers in gcloud ? resync is only happening for k8s services corresponding to those loadbalancers , right ? Or may be you are saying that if a k8s services is linked to a gcloud loadbalancer, the resync enables the controller code to go and check gcloud if the loadbalancing is provisioned or has anything changed there ? We need to document this in somwhere much like the apiconventions doc
- I mean loadbalancer in GCP as an example (but can also be other load balancer, persistent volumes in cloud or things like that). Yes - we resync on k8s objects, but that that let's you sync those. As an example - imagine service of type LoadBalancer - if you're periodically syncing the k8s service object, this triggers the operation of syncing the load-balancer (and we have an ability to check if it exists, is healthy etc.). - hmm, I would need to double check it, but I thought the default resync period is pretty long...
The default resync period is 12~24 hours. It is long but the bad experience will still happen every day.


Question: glog.V(3).Infof("Attempting to schedule: %+v", pod) start := time.Now() <del> defer func() { metrics.E2eSchedulingLatency.Observe(metrics.SinceInMicroseconds(start)) }() </del> dest, err := s.config.Algorithm.Schedule(pod, s.config.NodeLister) <del> metrics.SchedulingAlgorithmLatency.Observe(metrics.SinceInMicroseconds(start)) </del> if err != nil { glog.V(1).Infof("Failed to schedule: %+v", pod) s.config.Recorder.Eventf(pod, api.EventTypeWarning, "FailedScheduling", "%v", err) s.config.Error(pod, err) return } <ins> metrics.SchedulingAlgorithmLatency.Observe(metrics.SinceInMicroseconds(start)) </ins> b := &api.Binding{ ObjectMeta: api.ObjectMeta{Namespace: pod.Namespace, Name: pod.Name}, Target: api.ObjectReference{

Currently scheduler latency are observed regardless of failure. For example, binding latency is like: This could be a potential confusion if latency spikes happen because of failure. We should better handle it like separating error case from normal
/cc
I completely agree - are you going to change this?
Yes. I will do it.


Question: invalid("spec", "validation", "openAPIV3Schema", "properties[map]", "x-kubernetes-validations[0]", "rule"), invalid("spec", "validation", "openAPIV3Schema", "properties[map]", "additionalProperties", "x-kubernetes-validations[0]", "rule"), } <ins> allMessageExpressionValidationsErrors := []validationMatch{ invalid("spec", "validation", "openAPIV3Schema", "properties[x]", "x-kubernetes-validations[0]", "messageExpression"), invalid("spec", "validation", "openAPIV3Schema", "properties[obj]", "x-kubernetes-validations[0]", "messageExpression"), invalid("spec", "validation", "openAPIV3Schema", "properties[obj]", "properties[a]", "x-kubernetes-validations[0]", "messageExpression"), invalid("spec", "validation", "openAPIV3Schema", "properties[array]", "x-kubernetes-validations[0]", "messageExpression"), invalid("spec", "validation", "openAPIV3Schema", "properties[array]", "items", "x-kubernetes-validations[0]", "messageExpression"), invalid("spec", "validation", "openAPIV3Schema", "properties[map]", "x-kubernetes-validations[0]", "messageExpression"), invalid("spec", "validation", "openAPIV3Schema", "properties[map]", "additionalProperties", "x-kubernetes-validations[0]", "messageExpression"), } </ins> tests := []struct { <del> name        string storedRule  string updatedRule string errors      []validationMatch </del> <ins> name                     string storedRule               string updatedRule              string storedMessageExpression  string updatedMessageExpression string errors                   []validationMatch </ins> }{ { <del> name:        "functions declared for storage mode allowed if expression is unchanged from what is stored", storedRule:  "test() == true", updatedRule: "test() == true", </del> <ins> name:                     "functions declared for storage mode allowed if expressions are unchanged from what is stored", storedRule:               "test() == true", updatedRule:              "test() == true", storedMessageExpression:  "'test: %s'.format([test()])", updatedMessageExpression: "'test: %s'.format([test()])", }, { name:                     "functions declared for storage mode not allowed if rule expression is changed", storedRule:               "test() == false", updatedRule:              "test() == true", // rule was changed storedMessageExpression:  "'test: %s'.format([test()])", updatedMessageExpression: "'test: %s'.format([test()])", errors:                   allRuleValidationsErrors, </ins> }, { <del> name:        "functions declared for storage mode not allowed if expression is changed", storedRule:  "test() == false", updatedRule: "test() == true", errors:      allValidationsErrors, </del> <ins> name:                     "functions declared for storage mode not allowed if message expression is changed", storedRule:               "test() == true", updatedRule:              "test() == true", storedMessageExpression:  "'test: %s'.format([test()])", updatedMessageExpression: "'test - updated: %s'.format([test()])", // messageExpression was changed errors:                   allMessageExpressionValidationsErrors, </ins> }, }

Noticed a copy & paste issue when reading through the CEL code. Basically here: This should be instead of: The impact of this issue should be that messageExpressions are always evaluated with the CEL environment of the compatibility version. But stored (i.e. pre-existing) messageExpressions should be evaluated with a CEL environment with the "max" version instead. For users of Kubernetes this means that messageExpressions deployed with one minor version of Kubernetes might not work anymore if they rollback e.g. by one minor version. messageExpressions should be treated the same way as rules in that regard. Deploy Kubernetes 1.30 Deploy CRD that uses messageExpression that is using Sets (which became available with Kubernetes 1.30 that is using compatibilty version 1.29) Rollback to Kubernetes 1.29 Update something in the CRD =CRD update will fail because messageExpression is evaluated with the wrong CEL environment (it should still work) No response <detailsAt least on Kubernetes main, but I would assume this isuse is there since CEL was implemented </details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc Will open a PR in a bit


Question: updated.ObjectMeta.Finalizers = append(updated.ObjectMeta.Finalizers, servicehelper.LoadBalancerCleanupFinalizer) klog.V(2).Infof("Adding finalizer to service %s/%s", updated.Namespace, updated.Name) <del> // TODO(87447) use PatchService from k8s.io/cloud-provider/service/helpers _, err := patch(s.kubeClient.CoreV1(), service, updated) </del> <ins> _, err := servicehelper.PatchService(s.kubeClient.CoreV1(), service, updated) </ins> return err }

PR service patch methods to Once there is a release with that PR included, the code in service-controller and legacy-cloud-providers/gce can be updated to use those. /area platform/gce /assign
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Created a PR to use these new functions in Service-controllers.


Question: fuzz "github.com/google/gofuzz" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" apidiscoveryv2 "k8s.io/api/apidiscovery/v2" apidiscoveryv2beta1 "k8s.io/api/apidiscovery/v2beta1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

Seeing data race flakes in unit tests: Appears to have just been in /sig api-machinery /priority critical-urgent /assign /cc
This code is mutating the package-level schemebuilder: I think it should be adding the conversions to the local scheme instead:
Fixed in (found several instances)
/triage accepted


Question: // /api/{version}/* // /api/{version}/* func (r *APIRequestInfoResolver) GetAPIRequestInfo(req *http.Request) (APIRequestInfo, error) { <del> requestInfo := APIRequestInfo{} </del> <ins> requestInfo := APIRequestInfo{ Raw: splitPath(req.URL.Path), } </ins> <del> currentParts := splitPath(req.URL.Path) </del> <ins> currentParts := requestInfo.Raw </ins> if len(currentParts) < 1 { return requestInfo, fmt.Errorf("Unable to determine kind and namespace from an empty URL path") }

The apiserver proxy rewrites links to the form "/proxy/ns/default/...": But this form stopped working recently, seemingly around The "/proxy/ns/default/..." form always returns "404: Page Not Found", similar to the problem seen in If you remove "/ns/default" from the URL, it works.
/cc
I can investigate this as well.
I see the issue, sending a PR.
The proxy path needs to be built with knowledge of the apiversion of the incoming proxy request, and it needs to be updated to use the knowledge from the restmapper. This is a little trickier than I have time for right now, but it should definitely not be hardcode appending the /ns form. Not clear why this is not also caught by our test case. I can try and look at this more in the AM unless someone else wants to take a look in interim. /cc
Let me steal this one from you, I think I can fix it.
Have at it. /cc me on your PR
Sorry, I see you've already got a PR out!
No problem. I would be happy if you finished my PR relative to the commit I started. It will be a few days before I can go back to looking at this issue. I think v1beta1/v1beta2 proxy urls in go-restful need to always but the namespace scope in the path as the ultimate fix. I do want the proxy to have access to the restmapper per my PR. ----- Original Message ----- From: "Daniel Smith" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Friday, February 13, 2015 4:56:19 PM Subject: Re: [kubernetes] apiserver proxy stopped accepting namespaces () Sorry, I see you've already got a PR out! Reply to this email directly or view it on GitHub:
Any updates ?
This is breaking monitoring since it requires the proxy. Bumping up the priority to P0.
Sorry for dropping this. Working on a fix.


Question: services: []*v1.Service{{Spec: v1.ServiceSpec{Selector: map[string]string{"bar": "foo"}}}}, rss:      []*apps.ReplicaSet{{Spec: apps.ReplicaSetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"foo": "bar"}}}}}, // We use ReplicaSet, instead of ReplicationController. The result should be exactly as above. <del> expectedList: []schedulerapi.HostPriority{{Host: "machine1", Score: 0}, {Host: "machine2", Score: 5}}, name:         "disjoined service and replica set should be treated equally", </del> <ins> expectedList: []schedulerapi.HostPriority{{Host: "machine1", Score: 10}, {Host: "machine2", Score: 10}}, name:         "disjoined service and replica set matches no pods", </ins> }, { pod: &v1.Pod{ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{"foo": "bar", "bar": "foo"}, OwnerReferences: controllerRef("StatefulSet", "name", "abc123")}},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: /kind bug What you expected to happen: Problem 1: Predicates filter nodes. Existing pods on those nodes will be not be counted when calculating max pods per (zone or node) resulting in imbalanced cluster. Eg: removes nodes which cannot . If any of the pods are already scheduled on this node, they are not considered when counting in . Problem 2: When there are 2 selectors(service and replication controller), it is sufficient to match any one selector for distribution. This creates imbalance ]. Pods from previous deploys matches and are counted when distributing pods across zones/nodes (Even though they do not match ) . These pods will be deleted. After the deploy completes, the cluster is imbalanced - by zone and/or pods per node. How to reproduce it (as minimally and precisely as possible): Problem 1: Get pods to get scheduled on nodes with very high cpu utilization [Existing cpu utilization + 1 new pod will result in almost equal to 0]. GeneralPredicate will drop this node after scheduling the first pod. This will lead to over utilization of an already loaded zone. If there are enough nodes like this one, it will create a pile on effect resulting in most pods scheduled in an already loaded zone. Problem 2: Have 2 selectors and deploy 30 pods across 3 zones. Twice. Anything else we need to know?: Environment: Kubernetes version (use ): /sig scheduling <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
re Problem 1: for per node, we only need to compare current nodes as they will share the same maxPods. for per zone, we need to account other nodes, e.g. because of resource predicates. re problem 2: a little about your case :(
Problem 1: : It does not matter if the denominator is all nodes or current nodes. One with higher utilization will still end up with a lower score. : It is critical that the denominator is all nodes. Else we are ending up with highly imbalanced cluster. Problem 2: Here is an example list of selectors.

All pods matching any one service selector are counted in So in our case, old pods are counted during scheduling. But they get killed during the deployment. If the maxSurge and maxUnavailable is high enough, you will end up with an imbalanced zone after deployment.
Gentle ping.
/assign bsalamat
Re: Problem 1 - You can also reproduce this by using pod anti-affinity. Zone score is calculated only based on nodes filtered by predicates. IMO zone score should be calculated using all nodes.
I will try to separate this into two PRs. Any other suggestions welcome.
I am working Problem 1. Looks like I have to open a new issue.


Question: bkpPortal := b.Portals <del> // create new iface and copy parameters from pre-configured iface to the created iface </del> <ins> // If the initiator name was set, the iface isn't created yet, // so create it and copy parameters from the pre-configured one </ins> if b.InitiatorName != "" { <del> // new iface name is <target portal>:<volume name> newIface := bkpPortal[0] + ":" + b.VolName err = cloneIface(b, newIface) if err != nil { klog.Errorf("iscsi: failed to clone iface: %s error: %v", b.Iface, err) </del> <ins> if err = cloneIface(b); err != nil { klog.Errorf("iscsi: failed to clone iface: %s error: %v", b.InitIface, err) </ins> return "", err } <del> // update iface name b.Iface = newIface </del> } // Lock the target while we login to avoid races between 2 volumes that share the same

What happened: When creating a block volume, the iSCSI storage plugin stores volume-related data in different directories: As a result, some data is not cleaned up once the volume is detached: How to reproduce it (as minimally and precisely as possible): These are the objects I used to reproduce this issue: pod- /sig storage CC
/assign


Question: pod *v1.Pod, nodes []*v1.Node, ) (framework.NodeScoreList, error) { // If no priority configs are provided, then all nodes will have a score of one. // This is required to generate the priority list in the required format if len(extenders) == 0 && !fwk.HasScorePlugins() {

/sig scheduling A way to check the scoring of each node to figure out why a node has a high scheduling priority. proposal 1: add some debug level logs to Kube-scheduler, so we can know node score on each scoring step. proposal 2: a simple way to decrease the priority of one node(in my case a node with more than 200 pods is with the highest scheduling priority for an unknown reason) proposal 3:? All new pods are scheduled to one of five nodes in my cluster. I have checked both CPU/memory requested usage and all five nodes are bare metal and running about 150 pods or 200 pods. Pods on node request 80-93% of the memory. We have a similar issue as Currently, I think is the easiest way.
This is useful. how do you suggest we log this information? note that there are two dimensions for each pod: plugin and node score. Also, we need to limit the log length, we can't simply log the score for each node in clusters with large number of nodes.
I prefer to log only the top 5 or 10 node scores in the log. (I have not read the code yet., will dig into it later.)
We actually merged something like this that I wrote in , but had to revert it because of performance issue in large clusters. That could be a good starting point My general approach was to 1) invert the already available to map (rather than , which it currently does) and then 2) log the top 3 highest scoring plugins for each node However discovered performance issues after this in (we believe due to the many writes this process took) and it was reverted in I think we decided that some performance hit is reasonable, especially given log levels =4, but this was ~50% drop.
thanks for the information. So with current kube-scheduler, I can set log level to 10 to get every node score for each plugin. It helps in some cases.
yes, that's the current case. Though creates a lot of other noise in the logs, so I think we agreed that it is okay for us to add this kind of summary at a lower level like
I think we can use this chance to standardize the logging logic to dump scores dynamically depending on log level: for logging level ranged in , dump the scores for topM nodes/plugins.

And make it as performant as possible. for logging level ranged in , dump the scores for topN nodes, and probably show all plugins for logging level = 10, dump the scores for all nodes/plugins.
/assign
/triage accepted
/reopen Some problems that were mentioned by are not fixed yet. /cc
Reopened this issue. <detailsIn response to : /reopen After taking a glance, I can spot several issues: A consensus on the value and . This is not fully discussed yet. Further dig into the performance result. The test runs on v=1 and v=4, but actually they're the same if it's running in the context of integration framework. I'd expect to see the result comparing before/after on v=4 and v=6. Further review on the code, like adding UTs and some logic nits. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Since is merged to revert the change,
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Just FYI: this may be related to in the future.

In , we cannot add all scores to the event message (because the event msg has the limit on the length), so I am thinking we need to sort node-scores by total score and add the score of top N nodes to the event msg. And, if we do so, we can easily achieve to log the top M Node's score by using this sorted node-scores.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/lifecycle rotten
/remove-lifecycle rotten
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: kube::build::docker_image_exists golang 1.3 || { [[ ${KUBE_SKIP_CONFIRMATIONS} =~ ^[yY]$ ]] || { echo "You don't have a local copy of the golang docker image. This image is 450MB." <del> read -p "Download it now? [y/n] " -n 1 -r </del> <ins> read -p "Download it now? [y/n] " -r </ins> echo [[ $REPLY =~ ^[yY]$ ]] || { echo "Aborting." >&2

Sometimes it is nice to have non-tar'd binaries for releases. We should upload those along with the our packaged tarballs. This will help fix


Question: } func (recorder *recorderImpl) AnnotatedEventf(object runtime.Object, annotations map[string]string, eventtype, reason, messageFmt string, args ...interface{}) { <del> recorder.generateEvent(object, annotations, metav1.Now(), eventtype, reason, fmt.Sprintf(messageFmt, args...)) </del> <ins> recorder.generateEvent(object, annotations, eventtype, reason, fmt.Sprintf(messageFmt, args...)) </ins> } func (recorder *recorderImpl) makeEvent(ref *v1.ObjectReference, annotations map[string]string, eventtype, reason, message string) *v1.Event {

is defined here but never used. As this is an unexported method it doesn't affect anyone, just a small point of confusion for the reader. /sig api-machinery /kind bug
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted </details
/triage accepted removing triage since the PR seem to be fixing the issue /assign Please close when merged! thank you


Question: func NewFakeOOMAdjuster() *OOMAdjuster { return &OOMAdjuster{ <ins> pidLister:                 func(cgroupName string) ([]int, error) { return make([]int, 0), nil }, </ins> ApplyOOMScoreAdj:          fakeApplyOOMScoreAdj, ApplyOOMScoreAdjContainer: fakeApplyOOMScoreAdjContainer, }

The type has a field called to indicate that a container must run as non-root user. The description for this field includes the sentence , which would mean that setting the field to or to (unset) have the same effect: However, the PodSecurityPolicy provider implementation handles the (unset) case differently, and thus does differentiate between unset or values: The API type documentation should be updated to clarify this behaviour and reflect the actual PodSecurityPolicy implementation. /sig node
Thanks for reporting this issue. I think that would be best able to assist with this. /sig auth /remove-sig node
The pod documentation is correct. If the field is not set or is false, the enforcement is not performed by the kubelet. However, podsecuritypolicy with a runasuser strategy of mustrunasnonroot will force a pod that has not specified a runAsNonRoot setting or a runAsUser uid to runAsNonRoot: true The documentation on the MustRunAsNonRoot strategy of podsecuritypolicy should clarify the mutation behavior of the PSP /remove-kind bug /kind documentation
Can I take this issue?
^ this is something that can be changed.
Here's a proposed change to the documentation in : Is this sufficient to solve this issue? If so, should I submit a PR there?
I'd change that to to be a little more complete.
/assign See PR for docs at
GitHub didn't allow me to assign the following users: josiahbjorgaard. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign See PR for docs at Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
~This would have saved me some battling today! I couldn't work out why my pod wasn't selecting my run-as-root pod security policy and ignoring my restricted pod security policy - I didn't realise I had to explicitly set to get the correct PSP to match (I thought the restrictive PSP would reject the pod and then the run-as-root PSP would match, not that the restrictive PSP would mutate the PodSpec)~ It would have had I read the actual docs better sigh.


Question: ready kubernetes %v in-addr.arpa ip6.arpa { pods insecure <del> upstream </del> fallthrough in-addr.arpa ip6.arpa ttl 30 }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: I believe this started with Any thoughts /cc /priority important-soon /milestone v1.19 /sig network
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which test(s) are failing: /priority important-soon /milestone v1.19 /sig network </details
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: func diskConsumingPod(name string, diskConsumedMB int, volumeSource *v1.VolumeSource, resources v1.ResourceRequirements) *v1.Pod { // Each iteration writes 1 Mb, so do diskConsumedMB iterations. <del> return podWithCommand(volumeSource, resources, name, fmt.Sprintf("i=0; while [ $i -lt %d ];", diskConsumedMB)+" do dd if=/dev/urandom of=%s${i} bs=1048576 count=1; i=$(($i+1)); done; while true; do sleep 5; done") </del> <ins> return podWithCommand(volumeSource, resources, name, fmt.Sprintf("i=0; while [ $i -lt %d ];", diskConsumedMB)+" do dd if=/dev/urandom of=%s${i} bs=1048576 count=1 2>/dev/null ; i=$(($i+1)); done; while true; do sleep 5; done") </ins> } // podWithCommand returns a pod with the provided volumeSource and resourceRequirements.

Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: improved disk eviction testing to make it more precise. For example, it expects a pod which uses 299 Mb of disk not to be evicted if it sets its limit to 300 Mi. As shown on , this test passes for all OS distributions except CoreOS. I have ruled out an issue with the storage driver and backing filesystem since both COS and CoreOS use overlay backed by extfs. What you expected to happen: I expected the pod using 299Mb not to be evicted when its limit is 300 Mb. I expected not to see this event: How to reproduce it (as minimally and precisely as possible): The container runs this command to use 299Mb: Each iteration writes 1Mb to disk, producing a total of 299 Mb of disk usage. The pod has set to 300Mi. The test has the LocalStorageCapacityIsolation feature gate enabled. Anything else we need to know?: This may be a discrepancy in the (which we use to measure container disk usage) or , which is used by the test to write exact quantities of disk.
/sig node
cc can you take a look at this after thanksgiving?
I bumped the coreos image... going to see if it clears up the flake
It appears the root filesystem is being included in the disk space calculation . Do you think the coreos image might be slightly larger than the other images under test?
One more code snippet that is interesting:
thats the container's root filesystem
correct. the feature appears to add and compare to the quota: The reproducible steps do not take into account the rootfs space. If the quota is set to 300Mi, then the under loop count would have to take into account the rootfs space to not exceed the limit. I do not see anywhere in the tests or the reproduce steps this part of the calculation.
The refers only to the writable layer of the container, and excludes base (image) layers. With the provided command, the only files that are in the container's rootfs are the ones written by the command, and should amount to 299Mb. Additionally, the container does not write any logs (through stdout), so that value should be 0.

I ran a quick test on a vm with the current image, and found that both and correctly wrote and measured disk space correctly. I also tested as it inside a container, and verified that we are measuring the container's writable layer correctly. My guess now is that it has something to do with how we calculate disk usage by logging with journald. I am looking into it
Actually, the previous comment was incorrect. When I ran the test, I found that each file was 1028 kb instead of 1024 kb, which resulted in the command writing 300.3 Mb instead of 299 Mb. I will change the test to require slightly less precision. This appears to be because an empty file in coreos takes up 4 kb. Test this by running . On other OS we test on, empty files use 0 kb. This is what accounts for the difference.
I tracked down an issue to the dd call. The stdout and stderr need a redirect to /dev/null to suppress the following messages: Problem Line: Fix:
I PRed that fix and changed it to stderr, since the other tests seem to use stderr.
leaving this open until we get a green run
I'll continue to test locally to see if I find anything else.
opened PR with the help of and CoreOS defaults to using the actual size instead of the apprent size.


Question: // Starts background goroutines. If config_path, manifest_url, or address are empty, // they are not watched. Never returns. <del> func (kl *Kubelet) RunKubelet(config_path, manifest_url, etcd_servers, address string, port uint) { </del> <ins> func (kl *Kubelet) RunKubelet(config_path, manifest_url, etcd_servers, address, endpoint string, port uint) { </ins> if kl.DockerPuller == nil { <del> kl.DockerPuller = MakeDockerPuller() </del> <ins> kl.DockerPuller = MakeDockerPuller(endpoint) </ins> } updateChannel := make(chan manifestUpdate) if config_path != "" {

When using or with a key/value that has a multi-line string as value, that value will appear as quoted string in the log message, i.e. the log message will still be a single line. It would be nicer to print the value in text mode so that it goes across multiple lines, with clear indicators for start and end of the value. Indention could be used to avoid mistaking some embedded line as the start of a new log message. In JSON mode, the value should be formatted normally. /sig instrumentation /wg structured-logging Always printing as a single line can can make the log output unreadable ().
When adding something like this, can be reverted and implemented differently.
My original thinking was that only objects that explicitly ask for multi-line output get formatted that way. But that means that we need to review log calls very carefully. An alternative is to automatically switch from quoting strings to multi-line output based on some heuristic, for example when there's more than one line break.
Here's a potential implementation:
If structured logging is needed in any other case I can contribute, please inform if needed.
/triage accepted /assign /priority important-soon /area logging /cc /cc
We have the same problem as in kube-scheduler also in kubelet, it's just that no-one has seen and/or complained about the multi-line strings. With the benchmark + log analysis tool from I found the following multi-line string parameters in a kubelet log: <details</details


Question: --secure_port=6443  --portal_net=10.100.0.0/16  --etcd_servers=http://127.0.0.1:4001  <del> --public_address_override=$private_ipv4  </del> <ins> --public_address_override=${DEFAULT_IPV4}  </ins> --logtostderr=true Restart=always RestartSec=10

Minor issues with 1) The examples rely on cloud vendor specific variables ($private_ipv4 ) 2) The wget command for kube-register always re-downloads


Question: type ContainerCommandRunner interface { RunInContainer(containerID string, cmd []string) ([]byte, error) } <del> func GetUnusedImages(client DockerInterface) ([]string, error) { // IMPORTANT: this is _unsafe_ to do while there are active pulls // See https://github.com/docker/docker/issues/8926 for details images, err := client.ListImages(docker.ListImagesOptions{ Filters: map[string][]string{ "dangling": {"true"}, }, }) if err != nil { return nil, err } result := make([]string, len(images)) for ix := range images { result[ix] = images[ix].ID } return result, nil } </del>

I've been struggling on a problem over the past week, that a few pods always fail to run. When I tried to manually run one of those, I got: I could succesfully run the image on another os with different docker version and different backend filesystem, so I did many tests, to conclude that the problem is neither with btrfs, nor with docker, nor with coreos. I could create another aws autoscaling group, reusing the same versions of those tools, but without kubernetes, and I could sucessfully run that image there as well. I then removed all components from my cloudformation template, and readded them progressively, to notice that pulling this image only fails when the kubelet is running. I then tried to run the kubelet with different options, but the bare minimum is enough for docker to fail on that image. Looking at the traceback: Does kubelet try to query when only and are available? I still don't understand what does kubelet has to do with the btrfs error raised by docker, but the evidence is that everytime I stop kubelet, docker has no problem with the image: This issue is a blocker. Since I cannot get the cluster-side logging to work yet (), I'm trying to have a temporary fluentd pod running for application-side logging at first. Any suggestion?
Could this and be related? If so, think this should be p0. Who feels best equipped to dig on this one? , Adrien Lemaire wrote:
Yes, I believe it's the same issue :)
Maybe not related, but CoreOS moved from btrfs to overlay, see kubernetes issues -- sorry I don't have a CoreOS issue.
is most likely person to be able to know more about this issue.
If it happens again, it could help if someone could grab for all kubelet and dockerd threads (). That would allow us to determine if anyone is currently blocked in the kernel in a devicemapper-related system call.
Which version of kubernetes / kubelet are you running?
Ok I could re-produce the issue on aufs from HEAD: when kubelet is running: After I stopped kubelet process, every runs fine. I am going to mark this as P0, and assign it to myself.
Ok, I think I know the root cause of the issue here. Actually it is indeed caused by Image GC collection () we introduced recently.

Current support of gc of image depending on docker's "dangling=true", which gives us a wrong set unused images on a node. Instead it only returns a list of image layers without tag. When a image is pulling, all layers are untagged initially, then tagged later. In this case, some intermediate layers might be removed by kubelet GC, especially with a big image. We fixed the similar issues by adding lock during docker image pull triggered by kubernetes. But there is protection for docker image pull outside kubelet. To prove my hyperthesis, I add some debugging line to Kubelet, then run From below, one can see the same image layer is being removed. I did reopen to re-do Image GC, but didn't treat it as the highest issue in my queue. Now with these issues being reported, I think our current image GC does nothing except introducing issues, and we should remove it or completely disable it: 1) Issues reported like this. And it is very hard, or unnecessary complicated to fix this issue. 2) I think when we decided if a image should be GC, it should deal the entire image stacks, not one or two image layers. Even we might fix all race issues today, but we might break the logic later. Removing one or two intermediate layers is very hard to debug. 3) Since both docker and kubernetes pro all images having tag now. Any images with proper tag wouldn't be removed at all even they are not referred by any container for long time. cc/
is trying to disable garbage collection temporarily.
thank you !
I got this error on CoreOS, current stable (at this date) on DigitalOcean, for a single container that I had never built before -- rebuilding it after changing a single file making up the image, made the problem go away.


Question: client := newClient(true) i := 1 err := wait.PollImmediate(100*time.Millisecond, 5*time.Second, func() (done bool, err error) { <del> result := doer.Do(client, func(httptrace.GotConnInfo) {}, fmt.Sprintf("/echo?message=attempt-%d", i), 100*time.Millisecond) </del> <ins> result := doer.Do(client, func(httptrace.GotConnInfo) {}, fmt.Sprintf("/echo?message=attempt-%d", i), time.Second) </ins> i++ if result.err != nil {

https://prow.ppc64le- Tests in are flaking when run with master golang on ppc64le. After the commit No response The request to APIServer is timing out at The tests are Passing when the timeout value is increased to 200ms The PASS after increasing timeout: Seeing this falkiness only on architecture and when run with golang versions after the commit flakiness /sig testing
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : ### Which jobs are flaking? ### Since when has it been flaking? ### Testgrid link The PASS after increasing timeout: ### Anything else we need to know? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i can't seem to find the commit, which PR is that from?
Sorry for the confusion. It is the golang commit
has this an impact on https ? can this be a regression for performance and scalability in Kubernetes? /cc
how close were ppc64le tests to the 100ms timeout before? linux and darwin is nowhere remotely close to the timeout on go1.19 or go master with this diff: there's a ~tiny slowdown of the dial on linux, and a small slowdown on darwin: what does that show before/after on ppc64le?
cc on the question about existing ppc64le timing in
cc - are we running scalability tests on go tip? did we see any increases in CPU or latency or decreases in throughput in the last two weeks?
cc
we run scalability tests on go tip (using fixed version of K8s) but only using x64 architecture for masters and nodes in the cluster. We're also using Kubemark for scale testing go tip in K8s so the results might not be vocal enough about the regression.

No visible change in the pod throughput in our tests: ! I looked at some of the most popular <resource, subresource, scope, verbtuples for API call latency we measure in that test but I don't see any visible change at all as well. There might be an increase in apiserver CPU usage judging by the chart below but I'm not sure: ! That said, I'll edit the performance dashboard's config to include more runs to see whether the CPU usage bump is indeed a relatively fresh thing or not.
These failures occur only when run with flag. Without flag, as you said, it is nowhere remotely close to the timeout even on . But with the -race enabled, it is taking ~200ms on after the commit With the just previous commit , it took between 27ms -35ms
Do these tests run with -race flag enabled?
This is an end-to-end scalability test, not a test that one can run through . I also doubt we use the flag when building K8s binaries used in that e2e test given the overhead this flag has on the runtime - see the in article.
The tests this issue mentions are the unit tests run using the , which by default has flag while running a
Even on linux/amd64, I see a steep increase in time after the commit
definitely not, rewrites/instruments code so that it's performance is not at all representative of production use.
if the impact is limited to unit tests with race detection enabled, there's not a notable production impact, but I still asked if the impact was expected or will be optimized in https://go-
looks like there's also a revert CL open at https://go- in case it was needed for performance reasons, so I asked there if this level of impact to race detection code was expected
opened upstream
Meanwhile, Can we bump the timeout value to 200ms ? like we did in
that seems reasonable, since the point of this is not testing performance... though 200ms doesn't seem long enough given the comment in
True :( Will it be too much to ask for 250ms/300ms?
what timeout is used for this same request in other places in this test? it looks like a 1 second timeout is used
True, the same request is using a 1-second timeout at other places in this test!

As the same request is using 1 second timeout at other places, Can we increase it to 1s at
that seems fine to me.
xref
Can you PTAL


Question: framework.ExpectEqual(len(dsList.Items), 0, "filtered list should have no daemonset") }) <del> ginkgo.It("should verify changes to a daemon set status", func() { </del> <ins> /*	Release: v1.22 Testname: DaemonSet, status sub-resource Description: When a DaemonSet is created it MUST succeed. Attempt to read, update and patch its status sub-resource; all mutating sub-resource operations MUST be visible to subsequent reads. */ framework.ConformanceIt("should verify changes to a daemon set status", func() { </ins> label := map[string]string{daemonsetNameLabel: dsName} labelSelector := labels.SelectorFromSet(label).String()

[x] test approval issue : [X] test pr : 26 May [x] two weeks soak end date : 9 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining DaemonSet endpoints which are untested. Create a watch to track daemon set events that DaemonSet with a static label Create a daemon set with a static label. Confirm that the pods are running. Get the daemon set status. Parse the response and confirm that the daemon set status conditions can be listed. Update the daemon set status. Confirm via the watch that the status has been updated. Patch the daemon set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new which validates that three new endpoints are hit. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // need to verify that the item matches the set of scopes matchesScopes := true for _, scope := range options.Scopes { <del> innerMatch, err := scopeFunc(corev1.ScopedResourceSelectorRequirement{ScopeName: scope}, item) </del> <ins> innerMatch, err := scopeFunc(corev1.ScopedResourceSelectorRequirement{ScopeName: scope, Operator: corev1.ScopeSelectorOpExists}, item) </ins> if err != nil { return result, nil }

If a Pod is subjected to a with "PriortyClass" as its scope, the quota's calculation is incorrect. See repo steps for more details. with "PriortyClass" as its scope can be calculated at all times. Create a PriorityClass named "p1"Create a quota with as its scope. Note: w/o specifying , this issue won't happenCreate a Job and wait for its completion.Check quota's usage calculation Here is incorrect - it should be . Worse calculation may happen if I create a regular pod and then delete the Job:The eventual result is totally incorrect. No response master, and possibly all versions. kind <details</details<details</details<details</details<details</details
/sig api-machinery /kind bug
/triage accepted
I would like to take this up!
is already in flight for this


Question: return cmd, commandFns } <del> // makeSymlinks will create a symlink for each command in the local directory. func makeSymlinks(targetName string, commandFns []func() *cobra.Command) error { wd, err := os.Getwd() if err != nil { return err } var errs bool for _, commandFn := range commandFns { command := commandFn() link := path.Join(wd, command.Name()) err := os.Symlink(targetName, link) if err != nil { errs = true fmt.Println(err) } } if errs { return errors.New("Error creating one or more symlinks") } return nil } </del>

Follow up for the deprecation we did in 1.14:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle stale /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
I am willing to take this, if it is available :)
go for it. All you need to do is remove and associated code -
I ended up getting hit with the removal of the --make-symlinks flag. I looked at the threads linked to the original PR, but I did not find the actual reason behind the removal. Do you recall was the reason behind removing that capability?
the original request was from


Question: expectedList: []framework.NodeScore{{Name: "machine1", Score: 0}, {Name: "machine2", Score: 50}}, name:         "Another stateful set with partial pod label matches", }, <ins> { pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels:          labels1, OwnerReferences: controllerRef("StatefulSet", "name", "abc123"), }, Spec: v1.PodSpec{ TopologySpreadConstraints: []v1.TopologySpreadConstraint{ { MaxSkew:           1, TopologyKey:       "foo", WhenUnsatisfiable: v1.DoNotSchedule, }, }, }, }, pods: []*v1.Pod{ {Spec: zone1Spec, ObjectMeta: metav1.ObjectMeta{Labels: labels2, OwnerReferences: controllerRef("StatefulSet", "name", "abc123")}}, {Spec: zone1Spec, ObjectMeta: metav1.ObjectMeta{Labels: labels1, OwnerReferences: controllerRef("StatefulSet", "name", "abc123")}}, {Spec: zone2Spec, ObjectMeta: metav1.ObjectMeta{Labels: labels1, OwnerReferences: controllerRef("StatefulSet", "name", "abc123")}}, }, nodes:        []string{"machine1", "machine2"}, sss:          []*apps.StatefulSet{{Spec: apps.StatefulSetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"baz": "blah"}}}}}, expectedList: []framework.NodeScore{{Name: "machine1", Score: 0}, {Name: "machine2", Score: 0}}, name:         "Another stateful set with TopologySpreadConstraints set in pod", }, </ins> } for _, test := range tests {

Currently the scheduler by default spreads the pods of a deployment based on a hard-coded . We recently a pod object api (pod.Spec.TopologySpreadConstraints) that allows specifying spread behavior at the workload level. For pods that specify topology spread constraints, the default spreading behavior should not apply. /sig scheduling /cc
I can fix this behavior, can you assign it to me?
/assign sure, lets wait for feedback first.
sure, thanks
I agree. Having the default spreading run would just water down the score given by the constraints. Other than that, this would just be a temporary fix until is finalized


Question: serviceCIDRLister   networkingv1alpha1listers.ServiceCIDRLister serviceCIDRsSynced  cache.InformerSynced <del> readyCh chan struct{} // channel to block until the default ServiceCIDR exists </del> interval time.Duration }

Verify Master - Master Blocking verify.openapi-spec Inconsistently since 2 November. First failure (shown in testgrid) API Server is not becoming ready during Relevant logs /sig-testing
cc for cidr controller
and may be related.
/sig auth network
/remove-sig auth Is almost certainly related
/assign it seems the servicecidr controller takes time to start and it blocks the apiserver readiness
/reopen Test run failed after related PR merger:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yeah /assign
/triage accepted
this is sig network can you pelase update the title?
/retitle [Flaking Test][sig-network] ci-kubernetes-verify-master verify-openapi-spec
This still flake in master blocking test grid.
use 60s timeout and here uses 30s timeout by default. It flakes. 10 lines of log cannot show other details. My local test with 15s timeout failed several times. 30s is enough for my local tests. Or we just enlarge the timeout for CI.
having both scripts the same timeout sounds reasonable I'm still worried why it keeps timing out, but for that we should first define a test that measure the apiserver startup and then decide what value is correct and enforce it.
/priority important-soon /milestone v1.30 backport to v1.29 may be needed.
Hello! Release Signal shadow here. This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting (less than a month from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this issue is tagged for 1.30, is it still planned for this release?
This flakes in Do we have some action items for v1.30?
I need to fix it, is a shame on me, sorry
Hello! Release Signal shadow here. This issue has not been updated for a long time, so we'd like to check the status. The code freeze is starting less than a week, and we want to ensure that each PR has a chance to be merged on time.

As this issue is tagged for 1.30, is it still planned for this release? Please let us know the status if possible.
yeah, I like to get it fixed, there is some race condition that makes some of the boostrap to deadlock , I still didn't find why
didn't forget but didn't have time to figure out the root cause, will try to get back to this end of the week
I got one failure locally of an apiserver that took much time The apiservers need the servicecidr controller to start to be healthy, and the controllers need to sync the informer, but checking one of the requests from the informer I can see this error logging error output: "k8sx00nfnx02v1x12x06Statusx12x96x01nx06nx00x12x00x1ax00x12aFailurex1a2Timeout: Too large resource version: 2, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" and this keep going for a while logging error output: "k8sx00nfnx02v1x12x06Statusx12x98x01nx06nx00x12x00x1ax00x12aFailurex1a4Timeout: Too large resource version: 203, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" do you know why the resource version is growing?
looks like the server returns that error in a few places, not sure which one is getting exercised: update-openapi-spec starts with AllAlpha=true in order to enable all features / APIs ... are there any alpha features around watch / resourceVersion that could cause this?
Didn't read carefully the issue, so just quickly sharing some thoughts: 1) streaming lists might potentially be related, but it needs to be explicitly requested by the client.

Reflector is the only place that can currently do that, but it requires explicitly set env-var: Is it possible that this value is set? if so, then streaming lists will result is calling etcd for current RV and will wait for watchache to be synced with it - if progress-notify-interval is large and manual progress notify is disabled, we could face things like that ^^ 2) that said, if Alpha features are enabled, then manual-progress notify should also be set: 3) is it possible that we're facing the bug that Marek found 2 days before release that effectively was the main reason we didn't try to promote consistnet read to beta? ^^
Based on the log from I would guess this is the 3002ms implies the 3 second timeout of waiting for revision Empty implies consistent list from cache implies non-recursive list Planned to fix it, but PR was blocked on
thanks a ton for digging into that to deflake this test, I'd set this in the update-openapi- file (since it doesn't impact the generated openapi) with a comment to remove once is resolved


Question: return command } <ins> // calcNodeCidrSize determines the size of the subnets used on each node, based // on the pod subnet provided.  For IPv4, we assume that the pod subnet will // be /16 and use /24. If the pod subnet cannot be parsed, the IPv4 value will // be used (/24). // // For IPv6, the algorithm will do two three. First, the node CIDR will be set // to a multiple of 8, using the available bits for easier readability by user. // Second, the number of nodes will be 512 to 64K to attempt to maximize the // number of nodes (see NOTE below). Third, pod networks of /113 and larger will // be rejected, as the amount of bits available is too small. // // A special case is when the pod network size is /112, where /120 will be used, // only allowing 256 nodes and 256 pods. // // If the pod network size is /113 or larger, the node CIDR will be set to the same // size and this will be rejected later in validation. // // NOTE: Currently, the pod network must be /66 or larger. It is not reflected here, // but a smaller value will fail later validation. // // NOTE: Currently, the design allows a maximum of 64K nodes. This algorithm splits // the available bits to maximize the number used for nodes, but still have the node // CIDR be a multiple of eight. // func calcNodeCidrSize(podSubnet string) string { maskSize := "24" if ip, podCidr, err := net.ParseCIDR(podSubnet); err == nil { if ip.To4() == nil { var nodeCidrSize int podNetSize, totalBits := podCidr.Mask.Size() switch { case podNetSize == 112: // Special case, allows 256 nodes, 256 pods/node nodeCidrSize = 120 case podNetSize < 112: // Use multiple of 8 for node CIDR, with 512 to 64K nodes nodeCidrSize = totalBits - ((totalBits-podNetSize-1)/8-1)*8 default: // Not enough bits, will fail later, when validate nodeCidrSize = podNetSize } maskSize = strconv.Itoa(nodeCidrSize) } } return maskSize } </ins> // getControllerManagerCommand builds the right controller manager command from the given config object and version func getControllerManagerCommand(cfg *kubeadmapi.MasterConfiguration, k8sVersion *version.Version) []string { defaultArguments := map[string]string{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: For IPv6, the node CIDR size is hard coded to /64. This causes an error bringing up the kube-controller, as the size must be for a smaller subnet (larger value) that what is used for the pod subnet, which is specified for some CNI plugins (e.g. Calico). In addition, this is a catch-22, as the pod subnet prefix cannot be for a larger network (smaller value) than /66, so we cannot set it to /63, for example. What you expected to happen: The node CIDR size should be computed, based on the pod subnet specified, so that it can meet the requirements. How to reproduce it (as minimally and precisely as possible): Bring up a cluster using KubeAdm, specifying IPv6 and choose any pod subnet prefix desired. Anything else we need to know?: Note, there is a second requirement, that the difference between pod subnet size and node CIDR size be 16 or less (issue ). I'll be submitting a proposed PR to address this. Environment: Kubernetes version (use ): latest on master, using 1.9.2 in my case Cloud provider or hardware configuration: bare metal, single node OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux bxb-c2-77 4.4.0-109-generic -Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others: /area ipv6 /sig network
/assign pmichali


Question: } } <ins> func TestMatchingWithBoundVolumes(t *testing.T) { volumeIndex := NewPersistentVolumeOrderedIndex() // two similar volumes, one is bound pv1 := &api.PersistentVolume{ ObjectMeta: api.ObjectMeta{ UID:  "gce-pd-1", Name: "gce001", }, Spec: api.PersistentVolumeSpec{ Capacity: api.ResourceList{ api.ResourceName(api.ResourceStorage): resource.MustParse("1G"), }, PersistentVolumeSource: api.PersistentVolumeSource{ GCEPersistentDisk: &api.GCEPersistentDiskVolumeSource{}, }, AccessModes: []api.PersistentVolumeAccessMode{api.ReadWriteOnce, api.ReadOnlyMany}, // this one we're pretending is already bound ClaimRef: &api.ObjectReference{UID: "abc123"}, }, } pv2 := &api.PersistentVolume{ ObjectMeta: api.ObjectMeta{ UID:  "gce-pd-2", Name: "gce002", }, Spec: api.PersistentVolumeSpec{ Capacity: api.ResourceList{ api.ResourceName(api.ResourceStorage): resource.MustParse("1G"), }, PersistentVolumeSource: api.PersistentVolumeSource{ GCEPersistentDisk: &api.GCEPersistentDiskVolumeSource{}, }, AccessModes: []api.PersistentVolumeAccessMode{api.ReadWriteOnce, api.ReadOnlyMany}, }, } volumeIndex.Add(pv1) volumeIndex.Add(pv2) claim := &api.PersistentVolumeClaim{ ObjectMeta: api.ObjectMeta{ Name:      "claim01", Namespace: "myns", }, Spec: api.PersistentVolumeClaimSpec{ AccessModes: []api.PersistentVolumeAccessMode{api.ReadOnlyMany, api.ReadWriteOnce}, Resources: api.ResourceRequirements{ Requests: api.ResourceList{ api.ResourceName(api.ResourceStorage): resource.MustParse("1G"), }, }, }, } volume, err := volumeIndex.FindBestMatchForClaim(claim) if err != nil { t.Fatalf("Unexpected error matching volume by claim: %v", err) } if volume == nil { t.Fatalf("Unexpected nil volume.  Expected %s", pv2.Name) } if pv2.Name != volume.Name { t.Errorf("Expected %s but got volume %s instead", pv2.Name, volume.Name) } } </ins> func TestSort(t *testing.T) { volList := NewPersistentVolumeOrderedIndex() for _, pv := range createTestVolumes() {

E0513 22:55:45. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 266 E0513 22:55:45. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 266 E0513 22:55:48. 957 ] watch of api.Secret ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:55:48. 957 ] watch of api.Namespace ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:55:48. 957 ] watch of api.Secret ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested W0513 22:55:49. 957 ] Resetting endpoints for master service "kubernetes" to &{{ } {kubernetes default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 6443 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50.

957 ] Resetting endpoints for master service "kubernetes" to &{{ } {kubernetes default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 6443 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} E0513 22:55:50. 957 ] unable to sync kubernetes service: endpoints "kubernetes" cannot be updated: the resource was updated to 279 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 W0513 22:55:51. 957 ] Portal net unspecified.

Defaulting to 10.0.0.0/24. I0513 22:55:51. 957 ] Will report 10.0.0.22 as public IP address. E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:01. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:08. 957 ] watch of api.Secret ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:56:08. 957 ] watch of api.Namespace ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:56:08. 957 ] watch of api.Secret ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested FAIL FAIL 111.315s
Do we know which test is failing? The last messages in the log are from , but I don't actually see a test failure message printed, and I can't recreate any failures.

The "Unable to authenticate the request due to an error: Token has been invalidated" messages are expected since the test exercises attempting to use invalid tokens () Also, the "Stop requested" messages indicate that particular test completed and invoked the deferred stopFunc, which shuts down the controllers and closes the test server
Here is the failing test: --- FAIL: TestPersistentVolumeClaimBinder (3.45s) Expected claim to be bound: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:claim02 GenerateName: Namespace:default SelfLink:/api/v1beta1/persistentVolumeClaims/claim02?namespace=default UID:-f9c3-11e4-96ec- ResourceVersion:172 CreationTimestamp:2015-05-13 22:54:47 +0000 UTC DeletionTimestamp:<nilLabels:map[] Annotations:map[]} Spec:{AccessModes:[ReadOnlyMany ReadWriteOnce ReadWriteMany] Resources:{Limits:map[] Requests:map[storage:{Amount: Format:BinarySI}]} VolumeName:} Status:{Phase:Pending AccessModes:[] Capacity:map[]}}
you aware of the issue with TestPersistentVolumeClaimBinder?
I ran into above issue a lot.
Sorry, hadn't seen this sooner. Lost in the flurry of email traffic. I am looking into it now.
I cannot reproduce on F20, Go 1.3.1 I'll try 1.4 next. Any other info would be appreciated.
Here is the output of docker version from my desktop. Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.4.2 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.4.2 Git commit (server): OS/Arch (server): linux/amd64 I started to run into this issue two days ago, and just ran into it half hour ago right after I sync-ed to HEAD again. Any more information you need here?
Thanks :)
I cannot reproduce this on F20, Go 1.4.2 Any runtime settings I might be missing? I see hack/test- is set to "-race" .


Question: cmd := []string{"cmd", "/c", "echo windows-volume-test", ">", filePath} <ins> ginkgo.By("verifying that pod will get an error when writing to a volume that is readonly") </ins> _, stderr, _ := f.ExecCommandInContainerWithFullOutput(podName, containerName, cmd...) framework.ExpectEqual(stderr, "Access is denied.") } func doReadWriteReadOnlyTest(f *framework.Framework, source v1.VolumeSource, volumePath string) { var ( <del> filePath        = volumePath + "test-file" </del> <ins> filePath        = volumePath + "test-file" + string(uuid.NewUUID()) </ins> podName         = "pod-" + string(uuid.NewUUID()) pod             = testPodWithROVolume(podName, source, volumePath) rwcontainerName = containerName + "-rw"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- Kubernetes e2e suite: [sig-windows] [Feature:Windows] Windows volume mounts check volume mount permissions container should have readOnly permissions on hostMapPath Feb 23 14:59:47.591: FAIL: cmd: [cmd /c echo windows-volume-test C:tmptest-file], stdout: "", stderr: "Access is denied."
/sig windows
/assign
Looks like it's a flaky test instead of a failing test /cc
mentioned this could be related to permissions: There might be two things happening, the test isn't actually waiting for pod to start (which is why it passes sometimes) and but when it does start it fails the test due to the permissions
` is used for creating the pods. According to the comments, the Pods should be up and running:
On successful run, I see the following in the kubelet logs: I believe we're execing into the pod when the container is actually getting deleted by containerd but the pod is running which is not giving a correct indication of the command. I am modifying the test to ensure we would wait till all the containers are Ready and log the stdout and stderr to see if we're noticing issues. I tried manually running this test on a Windows node but couldn't reproduce it.
More examples of flaking runs:
/assign
I ran the test continuously: I am guessing this might be due to some of resources constraints we have seen when other tests runs at the same time (such as ). looking into it further...
Maybe we can convert it to a serial test case?
maybe, slowly looking into is some more... I find it interesting that which essentially does the same thing
Here's another:
Figured out what is happening: If the test runs before it leaves a file behind that the new test doesn't have permissions to. To repo: Then run:
/triage accepted


Question: # keep request = limit to keep this container in guaranteed class limits: cpu: 10m <del> memory: 20Mi </del> <ins> memory: 50Mi </ins> requests: cpu: 10m <del> memory: 20Mi </del> <ins> # Note that this container shouldn't really need 50Mi of memory. The # limits are set higher than expected pending investigation on #29688. # The extra memory was stolen from the kubedns container to keep the # net memory requested by the pod constant. memory: 50Mi </ins> args: - -cmd=nslookup kubernetes.default.svc.__PILLAR__DNS__DOMAIN__ 127.0.0.1 >/dev/null && nslookup kubernetes.default.svc.__PILLAR__DNS__DOMAIN__ 127.0.0.1:10053 >/dev/null - -port=8080

Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?


Question: // DecodeNestedObjects decodes plugin args for known types. func (c *KubeSchedulerConfiguration) DecodeNestedObjects(d runtime.Decoder) error { <ins> var strictDecodingErrs []error </ins> for i := range c.Profiles { prof := &c.Profiles[i] for j := range prof.PluginConfig { err := prof.PluginConfig[j].decodeNestedObjects(d) if err != nil { <del> return fmt.Errorf("decoding .profiles[%d].pluginConfig[%d]: %w", i, j, err) </del> <ins> decodingErr := fmt.Errorf("decoding .profiles[%d].pluginConfig[%d]: %w", i, j, err) if runtime.IsStrictDecodingError(err) { strictDecodingErrs = append(strictDecodingErrs, decodingErr) } else { return decodingErr } </ins> } } } <ins> if len(strictDecodingErrs) > 0 { return runtime.NewStrictDecodingError(strictDecodingErrs) } </ins> return nil }

With the addition of server side validation in 1.23, it is now possible for a call to successfully decode it’s data and return a non-nil error when there are strict decoding errors (e.g. duplicate/unknown fields). This means that callers of can distinguish between fatal errors (that cause decoding to fail) and strict decoding errors (that still allow the decode to succeed) by inspecting the target object (to see if it was successfully decoded into) and the error itself (to see if it is a ) With the NestedObjectDecoder however, it only returns an error from giving no indication whether a strict decoding error it has returned comes from an internal decode step that short circuited (by not accounting for strict decoding errors) or actually did successfully decode, but also encountered strict decoding errors. We propose taking a few actions to minimize the possibility of erroneous short-circuiting from Document how all nested decoders should only return strict decoding errors if all decoding is successful or return only non-strict decoding (ie. fatal errors). Document this for all known in-tree and out-of-tree . Update in-tree nested decoders to do the above mentioned behavior (which is just the kube-scheduler) Long-term solutions to better guarantee that no erroneous short-circuiting is taking place could include deprecating the current interface in favor of one that returns both the target object that was decoded into along with an error return a special error type specific to indicating that all internal decoding was successful.
/sig api-machinery
/triage accepted


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" <del> kubefeatures "k8s.io/kubernetes/pkg/features" </del> "k8s.io/kubernetes/test/e2e/framework" e2enetwork "k8s.io/kubernetes/test/e2e/framework/network" <del> e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" </del> imageutils "k8s.io/kubernetes/test/utils/image" admissionapi "k8s.io/pod-security-admission/api"

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Same failure in
I think this is related to the change @ could you please help take a look?
We already discussed this in The change from correctly points out that the test isn't working as expected. It needs to be fixed. /unassign
/assign
After , the tests are still failing. Could you help check it?
The skip function is used in another case as well. We may remove that like .
will fix this.
/close the test grid is green now.
Closing this issue. <detailsIn response to : /close the test grid is green now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: proxyTransportDial:         s.proxyTransportDial, serviceResolver:            s.serviceResolver, rejectForwardingRedirects:  s.rejectForwardingRedirects, <ins> tracerProvider:             s.tracerProvider, </ins> } proxyHandler.updateAPIService(apiService) if s.openAPIAggregationController != nil {

Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted


Question: "k8s.io/kubernetes/pkg/kubelet/util/format" hashutil "k8s.io/kubernetes/pkg/util/hash" "k8s.io/kubernetes/third_party/forked/golang/expansion" <ins> utilsnet "k8s.io/utils/net" </ins> ) // HandlerRunner runs a lifecycle handler for a container.

What happened: I tried to create a deployment with duplicate containerPorts with hostIPs of two different address families. Only the first mapping was provided as input into my CNI plugin chain. Also, a log message from kubelet warning about a duplicate containerPort was created. What you expected to happen: Both mappings to be provided as input into my CNI plugins. No log messages should be created. How to reproduce it (as minimally and precisely as possible): Create this deployment: Then grab kubelet logs on the host e.g: Anything else we need to know?: This issue here provides some context as to how I ran into this issue: Environment: Kubernetes version (use ): v1.15.1 Cloud provider or hardware configuration: DigitalOcean, custom setup OS (e.g: ):Kernel (e.g. ):Install tools: None Network plugin and version (if this is a network-related bug): Cilium v1.6.0 and Portmap v0.8.2 Others: N/A I've already identified the problem and will submit a PR which fixes this shortly. /sig network
is this dual-stack only? if so, does the fix for this need to target 1.16?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This applies to dual-stack only yes. I don't know if it should target 1.16, personally I'd like to get this out so I don't have to run a forked version of kubelet.
/assign
I would agree that this should hit v1.16 if possible.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


Question: -o "${tmp_kube_env}"  http://metadata.google.internal/computeMetadata/v1/instance/attributes/kube-env # Convert the yaml format file into a shell-style file. <del> eval $(python -c ''' </del> <ins> eval $(${PYTHON} -c ''' </ins> import pipes,sys,yaml for k,v in yaml.load(sys.stdin).iteritems(): print("readonly {var}={value}".format(var = k, value = pipes.quote(str(v))))

What happened: Pick an OS image where python is python3 and the cluster will not come up. What you expected to happen: Cluster should come up. How to reproduce it (as minimally and precisely as possible): Switch the default OS image to use an OS which has python3 as the default python. Run cluster/kube- for GCE. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/assign


Question: "type": "integer" }, "fsGroupChangePolicy": { <del> "description": "fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified defaults to "Always".", </del> <ins> "description": "fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used.", </ins> "type": "string" }, "runAsGroup": {

We need to move this field to beta in 1.20. /sig storage
/triage accepted
According to , is working on a better design on this.
Nope. The current design is the design we are going to use to handle fsgroup. we discussed this at length in sig-storage call.
Is it okay if I make this change?


Question: } count++ <ins> if len(output) > 0 && !shortOutput { return cmdutil.PrintResourceInfoForCommand(cmd, info, f, out) } </ins> cmdutil.PrintSuccess(mapper, shortOutput, out, info.Mapping.Resource, info.Name, dryRun, "created") return nil }

What happened: What you expected to happen: yaml output How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: Related PR


Question: proxier.deleteEndpointConnections(endpointUpdateResult.StaleEndpoints) } <ins> func (proxier *Proxier) acceptIPVSTraffic() { sets := []*IPSet{proxier.clusterIPSet, proxier.externalIPSet, proxier.lbIngressSet} for _, set := range sets { var matchType string if !set.isEmpty() { switch set.SetType { case utilipset.BitmapPort: matchType = "dst" default: matchType = "dst,dst" } writeLine(proxier.natRules, []string{ "-A", string(kubeServicesChain), "-m", "set", "--match-set", set.Name, matchType, "-j", "ACCEPT", }...) } } } </ins> // After a UDP endpoint has been removed, we must flush any pending conntrack entries to it, or else we // risk sending more traffic to it, all of which will be lost (because UDP). // This assumes the proxier mutex is held

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In a k8s cluster with IPVS proxy mode, when you create an ingress, the loadbalancer of this ingress can't be createautomatically, which caused the ipvs ci "gci-gce-ipvs" failed, link: http://k8s- The failed case is , with logs: Time out when waiting for for ingress to get an address. I'm not sure why, but I can see the pod keep restarting, And with container logs: AFAIK, ingress IP have nothing to do with kube-proxy, but when I change proxy mode to iptables, then ingress got it's IP in few seconds, though still in status. Is there any conflict between ipvs proxy rules and ingress controller? Maybe have this possibility. What you expected to happen: Ingress can got it's IP in ipvs proxy mode. How to reproduce it (as minimally and precisely as possible): a cluster in gce environment with proxy mode IPVS. . ingress with backends. for the ingress IP. TT Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): Ubuntu16.04.3 Kernel (e.g. ): 4.13.0-1008-gcp -Ubuntu SMP Thu Jan 25 11:08:44 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: cluster/kube-up Others:
/sig network /area ipvs
I found this maybe related with iptables rules , ingress-controller need to validate the default service by access . But the iptables like: That means the access of port will be directly DNAT to nginx-ingress-controller. In iptables proxy mode, DNAT will be done in chain, so the access of still functional. Thoughts?
I think the problem here is that the ingress controller is crash looping. You are correct that kube-proxy has nothing to do with the LB getting an ip address so I think we can rule out iptables / ipvs as the issue. I'm wondering why the nginx ingress controller in the test does not have permission to list objects in kube-system? Let me try reproducing and see what I find.

When you switched back to iptables mode, you say the container was still crashlooping. Was the error the same?
My mistake, actually the error log I pasted before is in iptables proxy mode, ipvs mode ingress-controller log are: I didn't notice the difference before. And now I'm sure the is created by kubenet, in host port map workaround. Problem is the access of is treated as , so match this rule:
So I'm confused. Why don't we see this problem in iptables mode then? What is the fix on the IPVS side?
I think that's because in iptables proxy mode, we don DNAT in , so traffic of service will never come to . Besides, iptables will match all virtual server IP in ipvs rules, make things even worse.
I think they are same issue.
/assign
Still get this issue with kubernetes 1.11.0. See below iptable rules in my host,


Question: // Close the queue. func (f *FIFO) Close() { <del> f.closedLock.Lock() defer f.closedLock.Unlock() </del> <ins> f.lock.Lock() defer f.lock.Unlock() </ins> f.closed = true f.cond.Broadcast() }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Race conditions found on 2 usages of sync.Cond in client-go/tools/cache.DeltaFIFO and client-go/tools/watch/eventProcessor. I don't know how harmful they are since the buggy interleaving is hard to trigger. They may lead to goroutine leaks to my understanding. They have similar causes and I put them in one issue here for ease of triage. When and are invoked concurrently on an empty queue, the in may miss the signal from . The is guarded by the for-loop below. It appears that should not be called if the FIFO queue is already closed, hence the check at line 475. However, the check above may race with : Specifically, when the queue is empty and not closed, will reach the . Between the check and , another goroutine can call and the broadcast on will be missed. Consequently, may block indefinintely and miss the broadcast on from . Note that holds while and hold . So can interleave with . Here is a stack trace found by our tool while running . The tool is based on the Go race detector. I did some research and found introduced the bug while trying to fix exactly the issue: "stop controller when the stop channel is closed and the queue is empty". The fix doesn't appear to solve the issue completely. One suggested fix: use in and instead of . This would enforce atomicity between the check and in . This is similar to DeltaFIFO. In between and : Another goroutine can call : This would cause the signal on to be missed by the above , leading to the goroutine being blocked unexpectedly. Stack trace from : One suggested fix: use mutex to protect in . What you expected to happen: No race condition. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): MacOS Kernel (e.g. ): Darwin Install tools: Network plugin and version (if this is a network-related bug): Others: This is my first issue filed.

Please let me know if I can provide further information.
/sig api-machinery
For issue 1, in probably the date race is not uncovered due to Run() being long running. If we use f.lock in f.Close() and f.IsClosed(), closedLock is not needed any more. Since f.lock is taken at the beginning of Pop(), we would need lockless version of IsClosed()
Oh..Right. I didn't see this point. Please ignore my suggested fix for issue 1 then.
/cc
Hi, I reproduced the goroutine leak with this test case. The control loop inside may not exit when is called on an empty queue due to the race condition. Also, I submitted a PR trying to fix it. :smile: