Question: type ByImageSize []kubecontainer.Image func (a ByImageSize) Less(i, j int) bool { <ins> if a[i].Size == a[j].Size { return a[i].ID > a[j].ID } </ins> return a[i].Size > a[j].Size } func (a ByImageSize) Len() int      { return len(a) }

What happened: When two or more images on a given node have the same size, their order inside the field set by the process is not deterministic. This problem has the side effect of continually triggering controllers which assume this list to be always equal on node updates unless images are or removed. What you expected to happen: The order of images should be stable even when two images have the same size, using the image ID as a second sort field would help this. How to reproduce it (as minimally and precisely as possible): Choose a node and run: Run a few times: It should be possible to see and images alternating on the output. Anything else we need to know?: Our particular use case make it really common for multiple images to have the same size on a given node. I came across this problem while debugging slow updates to DaemonSets and finding out that the kube-controller-manager was leaking memory until it eventually dies due to OOM: <img width="800" alt="Screen Shot 2019-06-30 at 20 27 22" src="https://user-"I was able to track this leak to a large number of watched elements in memory and by taking a look at the logs with increased verbosity I figured it was due to too many Node notifications being processed by the DaemonSet controller: <img width="800" alt="Screen Shot 2019-06-30 at 20 36 23" src="https://user-"It's possible that other controllers in the wild that watch nodes and only ignore the conditions timestamps on updates to face the same problem. I'll send a simple PR to fix this shortly. Environment: Kubernetes version (use ): But by taking a look at the code this problem also happens on master.
/sig node
Nice debugging here :)


Question: z/c	.	.	.	.	.	.	.	.	. */ <del> var _ = common.SIGDescribe("Netpol [LinuxOnly]", func() { </del> <ins> var _ = common.SIGDescribe("Netpol", func() { </ins> f := framework.NewDefaultFramework("netpol") ginkgo.Context("NetworkPolicy between server and client", func() {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Ive done preliminary testing with on calico powered clusters on some windows variants and found that the probing seems to work. Can we add wrapper to netpol suite so that we can run it on windows clusters ? Why is this needed: We need this because otherwise, CNIs such as EKS will give us This error is caused when, for example, pods that are landing on a windows node havent defined their target platform. Somehow i guess pods which are landing on linux nodes dont have this issue. Its also needed for the obvious reason that we probably want separate linux and windows test suites for policies, in general, even though we expect the logic to be identical (modulo hostports , which we dont currently test)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/assign
grepping around, I can't figure out where those are supposed to go. Can you give me pointers?
(resolved in slack, matts digging into this)
ok, these are running now on eks, lots of failures. will dig
/assign
the netpol are not being applied for the ns: Instead of failing because should not be able to access , the connection succeeds.


Question: ginkgo.By("Create a pod for further testing") hostBaseDir = path.Join("/tmp", ns) mountBaseDir = "/mnt/test" <del> basePod = f.PodClient().CreateSync(newHostPathTypeTestPod(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate)) </del> <ins> basePod = f.PodClient().CreateSync(newHostPathTypeTestPodWithCommand(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate, fmt.Sprintf("nc -lU %s", path.Join(mountBaseDir, "asocket")))) </ins> ginkgo.By(fmt.Sprintf("running on node %s", basePod.Spec.NodeName)) targetSocket = path.Join(hostBaseDir, "asocket") <del> ginkgo.By("Create a socket for further testing") _, err := utils.PodExec(f, basePod, fmt.Sprintf("nc -lU %s &", path.Join(mountBaseDir, "asocket"))) framework.ExpectNoError(err) </del> }) ginkgo.It("Should fail on mounting non-existent socket 'does-not-exist-socket' when HostPathType is HostPathSocket", func() {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Once we switched over the default runtime to containerd, the slow ci and pull jobs started timing out. The tests, which used to take 13 minutes, jumped to 30 minutes. This increase in test time is only observed in the slow job, so I suspect the slow job is exercising some functionality that may not be exercised as frequently in the other jobs. What you expected to happen: No performance regression How to reproduce it (as minimally and precisely as possible): Run the tests in the slow job with containerd Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
do we know what tests were regressing?
I did some analysis in , but basically, no single test stood out in terms of increasing the time took. I suspect that all the test cases started taking a few seconds longer, and the sum of all of contributed to the increase in job time.
I was mainly eyeballing the graphs, but we probably should get the raw data and do some more detailed analysis on it
do we have scale runs with the [slow] tests?
As an experiment, I changed the runtime back to docker in , and the slow job tests went back down to 13 minutes.
pointed out that may be relevant
/cc karan /cc
pointed out that cri-tools is not used in the kubelet path, so that particular issue may not be the cause. But maybe we can investigate image pulling as a first start. FWIW, there are at least 100+ test cases running in this job that create the same container on the same node multiple times: hostpathtype, multivolume, container restart
I think I found it. All the HostPath Socket tests are consistently never completing.
something might related
The test logs don't seem to show up in the artifacts. Running it manually, I see it hung on the exec during BeforeEach(): However when I do this manually using "kubectl exec" it is fine.

found a work around
i can easily recreate the problem locally using kubetest + local-up- running against containerd.
There are some notes on differences between how dockershim and other CRI impls handle this situation: have you run into this before?
Came up here:
I think what I said in that PR still applies.. I would expect kubectl -it to have a -d detach mode if background operation is the desired mode see docker exec -d option, and clarify what behavior is required of CRI exec sync.. up to 10sec pause for logging return value then return even if the in container exec process is still running? Note the hard coded start ops setting detach to false in docker shim: So it's using synch.. not detached.. but then later docker shim does this: before returning from the ExecSyncInContainer() request.
This seems to be a known issue. If an exec: 1) Spawns a long running background process; 2) Leaves the IO of the background process open after exits; 3) Containerd will wait until that IO to be closed before finishing the exec. It is arguable a bug in containerd, and the behavior of background process spawned by exec was never clearly defined. We've seen several production issues caused by this. Maybe we should: 1) Clearly define and document the behavior, and fix the test; 2) Pass timeout for probe (not related to this issue, but to protect the system). timeout is supported by many runtimes, but kubelet doesn't pass a proper value right now.
Among the callers of HandlerRunner#Run is the following from kubeGenericRuntimeManager#startContainer : We should find a proper value for the timeout.
We have a similar issue with CRI-O. I agree with that we should define this clearly and meanwhile setting a timeout makes sense.
I have opened to fix the tests.


Question: # Disable prompts from apt. ENV DEBIAN_FRONTEND noninteractive <ins> # Keeps unneeded configs from being installed along with fluentd. ENV DO_NOT_INSTALL_CATCH_ALL_CONFIG true </ins> RUN apt-get -q update &&  apt-get install -y curl &&

When I run Kubernetes cluster on GKE, it generates a lot of system logs coming from docker and kubelet. Even if my cluster runs just a few pods, I can see dozens or hundreds of log entries per second available in Google Cloud Logging in docker and kubelet logs. This makes the actual logs from my application get lost among them, if I do not set appropriate filter. Also, I am afraid that collecting massive amounts of logs imposes unnecessary workload on fluentd pod in my cluster. It would be good if I could just disable collection of kubelet/docker logs (and potentially other Kubernetes system logs) assuming that I do not want to have these logs exported.
/cc
since you tend to have a vested interest in node debug-ability, what's the minimum you'd want to have available here? From GKE's perspective, our users typically have no interest in these logs, so turning them off by default, even without an easy option to re-enable them, makes sense. Any user sophisticated enough to want them shouldn't have problems SSH'ing to the node for them. Is the story all that different in open-source kubernetes? Does anyone even care to have the kubelet and docker logs collected from the nodes? I've never used cloud logging to debug issues with them, it's always been easier to just SSH to the node. I'd lean towards not collecting them on the nodes in general. Also cc in case he has wisdom he'd like to impart
All I can say is that personally I have never accessed kubelet etc. logs "for real" when debugging -- as Alex says -- it is usually more direct just to SSH in. I would support not collecting these logs to improve cluster stability (but perhaps have a switch / technique to make it easy to collect them if the admin does care about this?).
One last call for to complain, because I'm starting on this now
Also
We want to avoid storing logs onto local disk in the future. In-order to achieve that, we need the existing cluster level logging infrastructure to work. Can the GKE problem be solved by moving system logs to a separate namespace? , Alex Robinson wrote:
How far in the future? Unless that's intended to happen soon, collecting the kubelet and docker logs in the meantime offers little value with a nontrivial resource (cpu+memory+egress) cost.

But yes, we can handle the system logs separately if we'd like. I think the most natural way would be to report them as compute engine logs, which could be done by adding an extra config option into the gcp output plugin.
, Alex Robinson wrote:
-- just to confirm, do you think this fix will be sufficient for you? In GKE, logs from containers will be listed under Container Engine, while logs from Docker and the Kubelet will be listed under Compute Engine, with the logs for each effectively being entirely seperated in the log viewer.
Yes, this will be sufficient in terms of not polluting Container Engine logs with system logs. I am still concerned about unnecessary load on fluentd caused by exporting Docker and Kubelet logs so ideally it would be great not to export these system logs if a user is not interested in them. But this is of slightly lower priority.
Hi, What's the status from this request? Is this available in 1.7.x or higher? Thanks.


Question: // according to whatever system is managing the endpoint. A nil value // indicates an unknown state. In most cases consumers should interpret this // unknown state as ready. For compatibility reasons, ready should never be <del> // "true" for terminating endpoints. </del> <ins> // "true" for terminating endpoints, except when the normal readiness // behavior is being explicitly overridden, for example when the associated // Service has set the publishNotReadyAddresses flag. </ins> Ready *bool // serving is identical to ready except that it is set regardless of the

The EndpointSlice API says and it works as expected in normal circumstances I delete the pod and the endpoints conditions are the expected However, if during this time I set we run into an incoherence with the API doc, but it seems that we already documented this edge case in the code I think that at this point is just to amend this information to the API, no? /kind bug /kind documentation
/sig network /assign
I would like to work on this. I just want a bit more clarification on what needs to be done!
I proposed a solution, that is clarify it in the doc ... once we reach an agreement in this issue we implement it ...
+1 to docs. I'd make it a little more vague, like: """ when the normal readiness behavior is being explicitly overridden, for example when the associated Service has set the publishNotReadyAddresses flag. """
so, the thing is to update the docs here with Tim's suggestion and here and do to update the generated code and submit a PR
/assign
I am not able to run command in windows, I am checking it. This error message is coming: *
/triage accepted
Is there a PR for this? Happy to review
I will raise the PR if my above problem gets solved. I am trying to make an update but the error mentioned above is coming.
What platform are you on? If the Makefile were really broken EVERYTHING would be failing.
he's on windows ... you will need a linux/macos. see
Let me know if you want someone to take it over.
I think in that case, you can assign this to someone else.


Question: import ( "io/ioutil" <ins> "strings" </ins> "testing" "time" <del> "strings" </del> "github.com/davecgh/go-spew/spew" "k8s.io/apimachinery/pkg/api/meta"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : let's say I run If the deployment is comes up in time it tells me which deployment is up, but if it times out it doesn't tell me on which deployment/resource. Could it be Why is this needed: That would make it easier to use, and the information provided is the same irregardless on whether the condition is met or not
/sig cli
Sounds reasonable for me, i can take care of this one if someone from owners will approve it as a feature.
/assign Zyqsempai
GitHub didn't allow me to assign the following users: Zyqsempai. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Zyqsempai Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: corelisters "k8s.io/client-go/listers/core/v1" "k8s.io/klog" "k8s.io/kubernetes/pkg/scheduler/algorithm/predicates" <del> "k8s.io/kubernetes/pkg/scheduler/algorithm/priorities" </del> "k8s.io/kubernetes/pkg/scheduler/framework/plugins/migration" framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" schedulerlisters "k8s.io/kubernetes/pkg/scheduler/listers"

/sig scheduling /priority important-soon See as an example. /assign
/unassign /help
/assign
/remove-help


Question: if !s.IsUnschedulable() { return nil, diagnosis, s.AsError() } <ins> // All nodes in NodeToStatusMap will have the same status so that they can be handled in the preemption. // Some non trivial refactoring is needed to avoid this copy. for _, n := range allNodes { diagnosis.NodeToStatusMap[n.Node().Name] = s } </ins> // Record the messages from PreFilter in Diagnosis.PreFilterMsg. msg := s.Message() diagnosis.PreFilterMsg = msg

When PreFilter returns UnschedulableAndUnresolvable, we don't need to run the preemption, but currently we run. This unexpected preemption could cause errors from the preemption by trying to read non-exist PreFilter data. As pointed out (), actually we didn't run the preemption in such cases in the past versions, but introduced this bug. /kind bug /kind regression /sig scheduling /triage accepted /priority important-soon
/assign
will fix this.


Question: return f.closed } <del> // Pop blocks until an item is added to the queue, and then returns it.  If </del> <ins> // Pop blocks until the queue has some items, and then returns one.  If </ins> // multiple items are ready, they are returned in the order in which they were // added/updated. The item is removed from the queue (and the store) before it // is returned, so if you don't successfully process it, you need to add it back

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In reviewing another PR, I noticed incorrect commentary and code. The primary offense is at , where a comment asserts (incorrectly) that a key can be present in but not and the code removes a key from without ensuring it is absent from . The only saving grace is that this code is actually unreachable (as also noted in the comment). I think it would be better to fix the comment and in this unreachable situation, than to violate an invariant that is critical to other code in this file. also contemplates the possibility that a key can be in and absent from . What you expected to happen: I expected there to be no comments or code that contemplate violations of the invariant at How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): current branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Sorry, I should have noticed this in my earlier pass over this code.
Agreed. Was discussing this with at
We discussed this one, Mike will send PR, Daniel will review.


Question: serviceAccountName: csi-hostpathplugin-sa containers: - name: hostpath <del> image: registry.k8s.io/sig-storage/hostpathplugin:v1.7.3 </del> <ins> image: registry.k8s.io/sig-storage/hostpathplugin:v1.9.0 </ins> args: - "--drivername=hostpath.csi.k8s.io" - "--v=5"

For the ReadWriteOncePod feature to graduate to beta, we need to have E2E test coverage for its current alpha behavior. See this section of KEP-2485 for more context on what cases we need to cover for alpha: This is a requirement for beta graduation, see:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage


Question: cpu: {{hollow_proxy_millicpu}}m memory: {{hollow_proxy_mem_Ki}}Ki - name: hollow-node-problem-detector <del> image: k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.7 </del> <ins> image: k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.8 </ins> env: - name: NODE_NAME valueFrom:

/cc Can and npd v0.8.8(with ) fix this problem? /sig node Example: node-problem-  /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
Yes, the and npd v0.8.8(with kubernetes/node-problem-detector) should fix this problem
/triage accepted /assign
There are other files that are needed to be updated, for reference Could you please update.


Question: deps = [ "//pkg/api:go_default_library", "//pkg/api/v1:go_default_library", <ins> "//pkg/apis/apps/v1beta1:go_default_library", </ins> "//pkg/apis/extensions:go_default_library", "//pkg/apis/networking:go_default_library", "//vendor/github.com/gogo/protobuf/proto:go_default_library",

Here are the hash label keys of all controllers: Deployment: DaemonSet: (new in 1.7) StatefulSet: (new in 1.7) Had a brief discussion with and about this, it should be consistent. We can use for DaemonSet and StatefulSet, and keep Deployment as-is. Deployment hash label key should be different from other two because Deployment pods hash labels is used by selector (unlike the other two controllers). We probably should have used hash annotations instead of hash labels for DaemonSet and StatefulSet, since we don't use hash labels to select their history or pods. Hash labels are chosen for consistency. This is fine since we don't modify hash label values.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Actually, controller revisions created by statefulset, daemonset has different label: // controller revision created by statefulset label: "" // controller revision created by daemonset label: controller-revision-hash: "" should be consistent?


Question: testPod.Status.Phase = api.PodFailed fakeWatch.Add(&testPod) <del> select { case <-received: case <-time.After(controllerTimeout): t.Errorf("Expected 1 call but got 0") } </del> <ins> t.Log("Waiting for pod to reach syncHandler") <-received </ins> }

Ran a test executable with more than 20,000 times and no failures. Any hints as to how to get it to show up? I'm running 100 jobs at a time, so my system is stressed already.
I to am unable to reproduce this. I get similar errors if I reduce the timeout at the top of the file. I spent a bit looking at how to rewrite the tests without the timeout, but nothing jumped out at me. Looking at it though, it's really just a 500ms timeout for the test to complete. I'm just going to remove them. Go has a default timeout of 10 minutes. If it's going to pass, it will pass then. If not, Go will give you a nice traceback where it deadlocked.


Question: stopCPU                  chan int stopMem                  chan int stopCustomMetric         chan int <ins> stopWaitGroup            sync.WaitGroup </ins> consumptionTimeInSeconds int sleepTime                time.Duration requestSizeInMillicores  int

https://k8s-
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
might be a culprit, but maybe not
This could be related to how the test initializes the framework or cleans it up, maybe.
Another failure in the same run, not just this one test:
I'm not positive but the stack trace might be wrong due to inlining optimizations. rc.framework == nil was my first guess, but that would fail earlier in that method.
Framework AfterEach runs namespace deletion, which may mean that .Namespace is nil (if inlining is indeed present).
I am looking into this right now.
It's unrelated. I reverted from the current master HEAD locally (via ), then did and finally and it did not help. The test still fails with the same message "Expected <*v1.Initializers | 0x0: nil not to be nil" (location: )
Note that I have executed one of the initializers e2e test (not autoscaling), because it is one of the tests reported by https://k8s- as panicked.
Looking a bit further I found out that commit the test above as already failing.
I have just realized that I have discovered yesterday a race condition in ResourceConsumer implementation, and might actually make it worse. Let me quickly make a PR out of a fix.
Let me try to recap what I have found out so far: The test that is being referred to at the top of this issue () fails "out of the box" and probably deserves a separate issue to track it Other failing tests that look like they have a similar failure pattern (NPE) fail because of trying to send a "consume CPU" request to a non-existent pod (unrelated to above, as far as I can tell) PR might cause the effect above (and probably does) because it actually deletes the pods that should have been deleted However, the root cause is that there is a race condition inside ResourceConsumer that makes "consume CPU" requests being sometimes sent already after "CleanUp" got called (fix prepared in PR )
Initializer was fixed in a different PR, depending on what cluster you test on.

, Andrzej Wasylkowski < :
Yeah, confirmed initializers test is now passing on GCE and disabled on GKE (which doesn't enable initializers configuration group yet) , Clayton Coleman <: The test that is being referred to at the top of this issue () fails "out of the box" and probably deserves a separate issue to track it Other failing tests that look like they have a similar failure pattern (NPE) fail because of trying to send a "consume CPU" request to a non-existent pod (unrelated to above, as far as I can tell) PR cause the effect above (and probably does) because it actually deletes the pods that should have been deleted However, the root cause is that there is a race condition inside ResourceConsumer that makes "consume CPU" requests being sometimes sent already after "CleanUp" got called (fix prepared in PR ) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
Clayton, can you assign this to me? I cannot do this myself.


Question: } } } <ins> type fakeEvaluator struct{} func (fakeEvaluator) Evaluate(a admission.Attributes) error { return errors.New("should not be called") } func TestExcludedOperations(t *testing.T) { a := &QuotaAdmission{ evaluator: fakeEvaluator{}, } testCases := []struct { desc string attr admission.Attributes }{ { "subresource", admission.NewAttributesRecord(nil, nil, schema.GroupVersionKind{}, "namespace", "name", schema.GroupVersionResource{}, "subresource", admission.Create, nil, false, nil), }, { "non-namespaced resource", admission.NewAttributesRecord(nil, nil, schema.GroupVersionKind{}, "", "namespace", schema.GroupVersionResource{}, "", admission.Create, nil, false, nil), }, { "namespace creation", admission.NewAttributesRecord(nil, nil, v1.SchemeGroupVersion.WithKind("Namespace"), "namespace", "namespace", schema.GroupVersionResource{}, "", admission.Create, nil, false, nil), }, } for _, test := range testCases { if err := a.Validate(context.TODO(), test.attr, nil); err != nil { t.Errorf("Test case: %q. Expected no error but got: %v", test.desc, err) } } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It appears that validating admission requests for server-side apply are populating the for requests to create Namespace objects. Non server-side apply requests don't do this, and a missing namespace field is the only way to distinguish cluster-scoped resources from namespace-scoped resources. This appears to be a breaking behavioral change. See What you expected to happen: Requests to create namespaces would have a consistent signature between server-side apply and non-server-side apply contexts. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: kind OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig-api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/wg api-expression
/assign
/assign
Requests made to namespaced URLs (URLs which contain a segment) populate the attribute in AdmissionReview requests. The documentation of the AdmissionReview field is as follows: From Update/Patch/Delete requests to specific namespaces have consistently populate this attribute for modifications to Namespace objects. Server-side apply is a type of patch request, so even when it is creating an object, that attribute will be populated. The difference is caused by a difference in the URL used by POST and apply-based creates. Creation via POST requests are made against the endpoint, so no attribute was populated. Namespace computed from request path: Namespace admission attributes for POST-based creates: Creation via server-side apply is done via a PATCH request to , so the attribute is populated. Namespace computed from request path: Namespace admission for apply-based creates: This is currently working as intended. If we want to make the behavior consistent between the two, I'd consider populating the namespace attribute for POST-based creation of Namespace objects.

Thanks for the thorough analysis Jordan, the problem OP is trying to solve is that they want to know if the object is namespaced or not, and used the presence of the field to detect that. According to your analysis, that seems wrong. The typical way to know if an object is namespaced is to use the discovery information, would that be the right way to solve that?
They could use discovery, or can continue to use the presence of the attribute, special-casing requests (namespaces are the only cluster-scoped resource which can have the attribute populated in AdmissionReview requests).
the documentation should be clearer, and I'd consider updating the POST-based create behavior to include this information to be consistent with update/patch/delete requests
do you want to update the doc and/or the post-based create behavior?
Sure! I am happy to take care of both.
where do you think it makes sense to add the namespace to the post-based behavior? We could add it to , but that might interfere with authorization, right? Would it be better to add the namespace directly to the ?
The info from the request body is not available when parsing requestinfo, only data in the URL. That's sort of what I expected, around the point we do this to pull the name from the incoming object:
Makes sense, I will give it a go, thank you!
that will also require sweeping admission plugins that check the namespace attribute. examples:
I think that we don't need to change anymore now. Do you know if there are other places in the docs where I should check?
yeah, I think we're good now. thanks for helping identify and clean this up


Question: deps = [ "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/version:go_default_library", </ins> "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/rest:go_default_library", ],

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: [] [sig-node] PreStop should call prestop when killing a pod [Conformance] Since when has it been failing: N/A Testgrid link: N/A Reason for failure: When spawning a Pod, it's container might start before all its networking plumbing has been completed. Thus, if the container tests / needs its network connectivity, it mgiht fail. The ` image tries to connect to the Kubernetes API, but if it cannot, the Pod will end up in an Error state. Retrying the Kubernetes API connection will avoid this issue. Anything else we need to know:
/sig node
/remove-sig node /sig testing


Question: i.interval = slice.get(6) i.transferBits = intOrFail("transfer port", slice.get(7)) i.bandwidthBits = intOrFail("bandwidth port", slice.get(8)) <del> return &i </del> <ins> return &i, nil </ins> } // StrSlice represents a string slice

What happened: The iperf tests were written a while ago but i think havent been updated. They measure bandwidth over multiple pods, but they probably should also have a test which reports individual pod's bandwidth between one pod over a SEP. They should pretty print a table so its easy to see exactly what is going on with the bandwidth They should use iperf3 (need to check that) Maybe we should have permutations for udp and tcp? The current output is something like: What you expected to happen: That running the tests would result in a table such as those that you can get from iperf3, i.e. How to reproduce it: run the e2es on any cluster, Environment: Kubernetes version (use ): master, any version
/sig network
/assign
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I've taken a first pass at this , working on resolving a couple issues then we can circle back to determine how else it could be improved!
Cool c/p the new test output so we can compare when u get a chance
ah yess. much better then what we have now.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: // volume types which support fsGroup based ownership(and permissions). // It will have no effect on ephemeral volume types such as: secret, configmaps // and emptydir. <del> // Valid values are "OnRootMismatch" and "Always". If not specified defaults to "Always". </del> <ins> // Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. </ins> // +optional FSGroupChangePolicy *PodFSGroupChangePolicy // Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported

We need to move this field to beta in 1.20. /sig storage
/triage accepted
According to , is working on a better design on this.
Nope. The current design is the design we are going to use to handle fsgroup. we discussed this at length in sig-storage call.
Is it okay if I make this change?


Question: // search for attachment by VolumeAttachment.Spec.Source.PersistentVolumeName attachment, err := c.k8s.StorageV1beta1().VolumeAttachments().Get(attachID, meta.GetOptions{}) if err != nil { <del> glog.Error(log("attacher.MountDevice failed while getting volume attachment [id=%v]: %v", attachID, err)) return err </del> <ins> return err // This err already has enough context ("VolumeAttachment xyz not found") </ins> } if attachment == nil { <del> glog.Error(log("unable to find VolumeAttachment [id=%s]", attachID)) return errors.New("no existing VolumeAttachment found") </del> <ins> err = errors.New("no existing VolumeAttachment found") return err </ins> } publishVolumeInfo := attachment.Status.AttachmentMetadata

/kind bug /sig storage What happened: When using a CSI driver that does not support NODESTAGE, I can see orphan json files in that are never cleaned up. What you expected to happen: No orphan file /assign


Question: // no old objects are left in storage), and then remove the rest of the // versions from this list. // Versions may not be removed from `spec.versions` while they exist in this list. <ins> // +optional </ins> StoredVersions []string `json:"storedVersions" protobuf:"bytes,3,rep,name=storedVersions"` }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, the field on must be explicitly set to their zero-values due to the fact that they are not tagged . Ref: Why is this needed: If using a system that automatically compares the provided configuration to the current, it will produce a diff on the updated status after creation. In addition, making sure to explicitly set to zero-values is not the best user experience. I am happy to provide the fix if this change is approved! /cc /sig api-machinery /area custom-resources
marking them as seems ok, so that client-side validation doesn't complain if they are absent when submitting manifests, but they shouldn't need to be set to omitempty (and at this point, we likely wouldn't change the serialization) Those fields are changed by controllers after creation, so expecting the live API object to exactly match the status is not going to go well anyway.
Good point! I’ll open a fix for this. Yep! The specific system in question ignores fields that are not set at creation time by default when doing comparison, so if status fields are not included then it will not report a diff.
/assign
I opened a fix for this but CC'ed both you and Let me know if any additional effort is required. Thanks!


Question: func TestGetDevicePrefixRefCount(t *testing.T) { fm := &mount.FakeMounter{ MountPoints: []mount.MountPoint{ <del> {Device: "/dev/disk/by-path/prefix-lun-1", Path: "/mnt/111"}, {Device: "/dev/disk/by-path/prefix-lun-1", Path: "/mnt/222"}, {Device: "/dev/disk/by-path/prefix-lun-0", Path: "/mnt/333"}, {Device: "/dev/disk/by-path/prefix-lun-0", Path: "/mnt/444"}, </del> <ins> {Device: "/dev/sdb", Path: "/127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-0"}, {Device: "/dev/sdb", Path: "/127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-1"}, {Device: "/dev/sdb", Path: "/127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-2"}, {Device: "/dev/sdb", Path: "/127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-3"}, </ins> }, }

During iscsi volume cleanup, kubelet reads from and retrieves a list of device paths where iscsi volume mounts on. The device path was expected to be in the form of . But currently the device path in is in the form of , thus iscsi panicked . I suspect this panic started to happen after switched to use from . seems to always resolve device path (see output below), and thus the expected iscsi configuration in the device path no longer exists.


Question: // requestWaitLimit comes from server configuration. requestWaitLimit time.Duration <ins> // watchTracker implements the necessary WatchTracker interface. WatchTracker // the most recent update attempts, ordered by increasing age. // Consumer trims to keep only the last minute's worth of entries. // The controller uses this to limit itself to at most six updates // to a given FlowSchema in any minute. // This may only be accessed from the one and only worker goroutine. mostRecentUpdates []updateAttempt </ins> // This must be locked while accessing flowSchemas or <del> // priorityLevelStates. lock sync.Mutex </del> <ins> // priorityLevelStates.  A lock for writing is needed // for writing to any of the following: // - the flowSchemas field // - the slice held in the flowSchemas field // - the priorityLevelStates field // - the map held in the priorityLevelStates field // - any field of a priorityLevelState held in that map lock sync.RWMutex </ins> // flowSchemas holds the flow schema objects, sorted by increasing // numerical (decreasing logical) matching precedence.  Every

Two flakes recently: They fail with error like: or /assign
/sig scalability
In logs I have found the following slow calls: It's a bug that we use exempt priority level for clusterloader: I created It's quite worrisome that APF handler takes 1s for exempt request.
I checked a APF code and I think the flow for exempt requests is: genericfilters.WithPriorityAndFairness() workEstimate := workEstimator.EstimateWork(r) fcIfc.Handle() cfgCtlr.startRequest() () looks for matching flowschemas in cfgCtrl.flowSchemas returns for exempt for other requests it calculates sha256 for flowDistinguisher and queues request req.Finish updates metrics calls execFn updates headers handler.ServeHTTP which stops the timer for priorityandfairness For me, the most likely reason for slowness is lock contention on We know that the test is super cpu-intensive in that part and other requests are e.g. calculating sha256 keeping that lock. A potential improvement would be to change this lock to RWLock or return the lock before starting calculating sha256 (if we believe that this is the most expensive operation there). /cc
/assign
I think the lock-contention is even worse than what you wrote, as e.g.: is happening under this lock (which is also dispatching requests). I will take a deeper look later this week, but I think that switching to RWLock might do the job without any invasive changes.
may trigger the execution of some requests that were waiting, but they are not executed in the same goroutine. Yes, I think changing that Mutex to a RWMutex would be an easy way to greatly reduce contention.
I agree switching to RWMutex is likely to help a lot, if that's not enough I think with a little work we could e.g. switch to s.
(perhaps because that's what my code originally did? Reviewers thought the retirement dance was too tricky.)
/triage accepted /priority important-soon I will work on the fix.
/sig api-machinery
Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped. /reopen
Reopened this issue. <detailsIn response to : Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped.

/reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, can you please report the filter latency regardless of whether it is bad or good? I would like to know how much of a difference that PR made.
Sure, will report the latency change in both cases.
I compared (the one where we saw issue) (first with the lock improvements) I tried to find requests that: use exempt PL (so shouldn't be throttled) has fl_priorityandfairness, key which means that priority and fairness filter took more than 100ms Sample matching request: In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! So overall it looks great. The only worrisome metric I see is that actual P&F throttling for other requests has grown: With locking improvements: ! Old: ! E.g. 'node-high' has now throttling latency of ~1.5s in spikes where in the previous test run it was more like ~800ms. I don't think the lock improvements touched this directly. I expect that this is either: per-test variance (if we rerun the old run again we would see different throttling latencies as well) we throttle exempt requests less (as expected) giving less resources to other requests. Anyway, I don't think this change is meaningful as per-test latency aggregates for PATCH nodes/status and PUT leases (the most common requests in node-high) look ~same as previously: ! ! I think we can close this for now as the issue seems to be resolved. Thanks a lot for your help here! /close
Closing this issue. <detailsIn response to : I compared In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! ! ! I expect that this is either: ! ! /close </details
: thanks for reporting!


Question: return &ResourceConfig{GroupVersionConfigs: map[schema.GroupVersion]bool{}, ResourceConfigs: map[schema.GroupVersionResource]bool{}} } <ins> // DisableAll disables all group/versions. It does not modify individual resource enablement/disablement. </ins> func (o *ResourceConfig) DisableAll() { for k := range o.GroupVersionConfigs { o.GroupVersionConfigs[k] = false } <del> for k := range o.ResourceConfigs { o.ResourceConfigs[k] = false } </del> } <ins> // EnableAll enables all group/versions. It does not modify individual resource enablement/disablement. </ins> func (o *ResourceConfig) EnableAll() { for k := range o.GroupVersionConfigs { o.GroupVersionConfigs[k] = true } <del> for k := range o.ResourceConfigs { o.ResourceConfigs[k] = true } </del> } // DisableVersions disables the versions entirely.

https://k8s- sample failure: (also affecting other CI jobs) cc /sig gcp
/assign We're interested in disabling the job as blocking entirely. We're of the opinion this shouldn't have bee allowed to block the community in the first place. I swear there's a historical issue around tightening the regex somewhere.
... wrong button sorry about that
possibly caused by erroring on invalid values that were previously silently skipped opened to downgrade that error to a warning for a release
Well, I mentioned the idea of making the job optional in
PR out to drop the runifchanged regex from pull-kubernetes-e2e-gke
Confirmed turns the test green
Can call this closed when merges
/pony hooray
Couldn't find a pony matching that query. <detailsIn response to : /pony hooray Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return cmdutil.AddSourceToErr("creating", info.Source, err) } <ins> // prune nulls when client-side apply does a create to match what will happen when client-side applying an update. // do this after CreateApplyAnnotation so the annotation matches what will be persisted on an update apply of the same manifest. if u, ok := info.Object.(runtime.Unstructured); ok { pruneNullsFromMap(u.UnstructuredContent()) } </ins> if o.DryRunStrategy != cmdutil.DryRunClient { // Then create the resource and skip the three-way merge obj, err := helper.Create(info.Namespace, true, info.Object)

As per , I'm trying to remove CPU limits. But the behaviour is different depending whether the ResourceQuota exists or not: I would expect to be idempotent, and for its behaviour to result in the same end configuration regardless of the initial configuration. a new namespace named this command: that the ResourceQuota is created and is set to it again that is removed each subsequent time it's run, as long as resourcequota/compute-resource exists resourcequota/compute-resource again that the ResourceQuota is created and is set to again, and so forth the exact same behaviour if is changed to No response <details</details<detailsOpenShift 4.13.12 </details<details</details<details</details<details</details<details</details
/sig node
Tried to replicate the issue and observed the same behaviour.
/remove-sig node /sig api-machinery /cc
What's the goal of applying here? What are you trying to achieve? I agree that this is not the right behavior though.
I'm trying to remove a previously set property. I cannot remove the ResourceQuota altogether because it contains other properties (which I left out of my example) that I do not wish to remove.
But 0 is the same as unset, isn't it?
No, if I set it to 0 then any deployment with set to higher than 0 will fail to create any pods because it will exceed the quota.
/triage accepted /cc
/assign
Hello, I'm having a similar issue. Here's my use case: We use Flux to deploy things on our cluster - in particular, we use the component and set chart values in . Some of the charts we deploy have default values we want to remove, so we want to set a value to Editing an HelmRelease with this kind of ajustments drops the null keys, so default values end up being setup again and generated manifests are invalid :( Let me know if you need anything else to keep this issue moving!
I think the correct behavior is to set the struct to the default value when or , because in the standard library does this as well, and I think we expect the result to be as similar as possible to . So our real issue here is that we didn't set the to its default value when patch.
Based on the above, I think this question is not related to the current issue.

Tracing it back to , maybe the current result is what we want. The explicit null values means that we want to remove it in patch. As mentioned , our behavior in create is also as expected. I'm not quite sure if we actually want to fix it, and how /cc I hope to get your advice as you are the implementer of
I think client-side on create should do what on update will accomplish, namely, explicit nulls in the manifest are treated as the removal of the attributes they are values of. Proof of concept opened at , I'd be happy for someone to pick that up, add test coverage, and work through any failing tests that change uncovers
Thank you, but I still have a question. If we solve the difference between create and patch behaviors in ApiServer on the client side, will third-party callers still have the same problem? Or should we not cover this scenario?
I would consider changing behavior of null in a create request server-side to be a compatibility break. values are defined for client-side apply to mean "remove the key associated with this value". I think fixing the client-side apply creation case to have the same semantic is the most narrowly scoped fix we can make while preserving compatible server-side behavior generally.
Okay, I got it, I'll follow up with


Question: } func (i *internalContainerLifecycleImpl) PreStopContainer(containerID string) error { <del> if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.CPUManager) { </del> <ins> if i.cpuManager != nil { </ins> return i.cpuManager.RemoveContainer(containerID) } return nil } func (i *internalContainerLifecycleImpl) PostStopContainer(containerID string) error { <del> if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.CPUManager) { </del> <ins> if i.cpuManager != nil { </ins> err := i.cpuManager.RemoveContainer(containerID) if err != nil { return err

Recently merged PR () broke clusters brought up with only GA APIs+features. See conformance test at /sig node /priority critical-urgent /kind bug /milestone v1.18 /assign


Question: "github.com/golang/glog" "github.com/xanzy/go-cloudstack/cloudstack" "k8s.io/api/core/v1" <ins> cloudprovider "k8s.io/cloud-provider" </ins> ) type loadBalancer struct {

/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?


Question: &CronJob{}, &CronJobList{}, &api.ListOptions{}, <ins> &api.DeleteOptions{}, </ins> ) scheme.AddKnownTypeWithName(SchemeGroupVersion.WithKind("ScheduledJob"), &CronJob{}) scheme.AddKnownTypeWithName(SchemeGroupVersion.WithKind("ScheduledJobList"), &CronJobList{})

Kubernetes version: HEAD What happened: Using interface to delete Job throws an error . What you expected to happen: Job should be deleted. We've found that issue after dependency update on dashboard: We are using to delete resources. It works fine for all other resources. I've investigated that and it appears to be the core issue. Other groups have registered as a known type. however batch is missing it After adding I was able to successfully delete Job using our code. I can quickly create PR for that. Just need a confirmation that this is indeed a bug. cc
+1 apis/autoscaling is similar
I thought we only needed to register it in the external versions (). I'll double check. cc
Registering api.DeleteOptions in batch looks fine. In the future, I think it's more proper to register DeleteOptions and ListOptions as unversioned, like how we treat .
DeleteOptions and ListOptions are expected to be in their group. They have to be in internal version and external version. Most of this code will be fixed in 1.6 (we'll standardize registering the "common" things). , Chao Xu wrote:
I've created PR to fix this issue. Could anyone check that?


Question: const ( imageWatcherStr = "watcher=" imageSizeStr    = "size " <del> sizeDivStr      = " MB in" </del> kubeLockMagic   = "kubelet_lock_magic_" // The following three values are used for 30 seconds timeout // while waiting for RBD Watcher to expire. rbdImageWatcherInitDelay = 1 * time.Second rbdImageWatcherFactor    = 1.4 rbdImageWatcherSteps     = 10 <ins> rbdImageSizeUnitMiB      = 1024 * 1024 </ins> ) func getDevFromImageAndPool(pool, image string) (string, bool) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: kubectl create -f ceph- and result is ok now i want to resize pv from 1G to 2G, edit kubectl describe pvc pvc-expand-test What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: ceph storageclass yaml: ceph secret: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T14:41:50Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration:OS (e.g. from /etc/os-release): CentOS Linux release 7.4.1708 (Core) Kernel (e.g. ): Linux m01 3.10.0-693.el7.x8664 SMP Tue Aug 22 21:09:27 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: ceph version 13.2.2 () mimic (stable) <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What's the version of your ceph cluster? k8s uses the command to get rbd image's size, and it expects the size unit in the output is . However, in your environment the unit is :
ceph version 13.2.2 () mimic (stable)
It seems that the output format is different between different ceph versions.

I will try to fix it.
/assign
your PR has been merged to master branch, and when will Kubernetes v1.11 fix this bug?
Sorry for the late, I've send cherry-pick PRs:
It seems this change is not working in some cases: I think this is due to two concurrent conditions: 1- ceph client still complain on stderr about not being able to load config file even if there is no need as a key is supplied and the result is (hence the output on stderr of ). This happens not only in hyperkube. 2- it seems we capture both stdout and stderr from the execution of the command (please note the output in the error from ) and then there is then a problem trying to unmarshall the JSON (to be clear, the unuseful warning is on stderr, the json output on stdout). I am able to have a workaround creating empty files for and so that the ceph client does not output anymore but I feel it would be much better to parse only stdout. A reference on the source code for Thanks edit:typo
could you tell where did you put the empty and ceph.keyring files? On the node running kubelet?
Inside the pod of the as the rbd resize is executed by this component.
I had the same problem, I ceph. conf and ceph. Keyring and solved it, but I think k8s got the default user of CEPH before going back to ceph. conf and ceph. Keyring. The following code validates my conjecture: //num 672 Ultimately, I want k8s to get those two files through configurable CEPH parameters. What do you think?
as far as I know you do not need to configure those two files if you use a key. That is still an hypothesis as in the version of Ceph we are using this switch seems to be undocumented: Anyway the warning can be safely ignored and I do not see why we should parse when there is no error
I two files with empty content before, no error; later I a soft connection called and ceph. Keyring (pointing to the ceph configuration file and key ring generated by my cluster), but ceph did not report an error. . I think ceph only needs two files to exist and doesn't care about its contents.

Fortunately, the current measures can solve the problem.
this matches my experience (see above, I created the empty files too and it gives no error) but another possibility is just not parse which I find "cleaner" as there is no error in this case even from the error code, and even what we get is just a warning.
Same issue as here. The warnings about the missing etc files break the JSON output. Adding empty files in the container fixes resizing.
Shouldn't this be reopened until the bug is fixed? Creating the two files also "solves" the issue for us.
/reopen cc
Reopened this issue. <detailsIn response to : /reopen cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
which version of ceph cluster is in use in your setup ?
We are running a ceph cluster with . kube-controller-manager is running which internally uses . Also just to prevent any confusion: I've used to resize the PVC, not rancher.
This also seems related to .
We are running
Are there plans to resolve this issue?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Answering myself - apparently it has been fixed in k8s 1.20:


Question: exit 0 } <del> if (-not (Test-DockerIsInstalled)) { Install-Docker } # For some reason the docker service may not be started automatically on the # first reboot, although it seems to work fine on subsequent reboots. Restart-Service docker Start-Sleep 5 if (-not (Test-DockerIsRunning)) { throw "docker service failed to start or stay running" </del> <ins> $kube_env = Fetch-KubeEnv Set-EnvironmentVars # Install Docker if the select CRI is not containerd and docker is not already # installed. if (${env:CONTAINER_RUNTIME} -ne "containerd") { if (-not (Test-DockerIsInstalled)) { Install-Docker } # For some reason the docker service may not be started automatically on the # first reboot, although it seems to work fine on subsequent reboots. Restart-Service docker Start-Sleep 5 if (-not (Test-DockerIsRunning)) { throw "docker service failed to start or stay running" } </ins> } Set-PrerequisiteOptions <del> $kube_env = Fetch-KubeEnv </del> if (Test-IsTestCluster $kube_env) { Log-Output 'Test cluster detected, installing OpenSSH.'

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Docker is always installed on node boot which adds significant start time and adds software that's unnecessary. containerd Windows nodes should not have Docker installed. Start a Kubernetes GCE Windows cluster via kube-up. You can run stuff like on it. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig windows /triage accepted


Question: "sigs.k8s.io/yaml" ) <del> func TestNoManagedFieldsUpdateDoesntUpdateTime(t *testing.T) { f := NewTestFieldManager(schema.FromAPIVersionAndKind("v1", "Pod"), "", nil) </del> <ins> func TestManagedFieldsUpdateDoesModifyTime(t *testing.T) { var err error f := NewTestFieldManager(schema.FromAPIVersionAndKind("v1", "ConfigMap"), "", nil) </ins> <del> obj := &unstructured.Unstructured{Object: map[string]interface{}{}} if err := yaml.Unmarshal([]byte(`{ </del> <ins> err = updateObject(&f, "fieldmanager_test", []byte(`{ </ins> "apiVersion": "v1", <del> "kind": "Pod", </del> <ins> "kind": "ConfigMap", </ins> "metadata": { <del> "name": "pod", "labels": {"app": "nginx"} </del> <ins> "name": "configmap" </ins> }, <del> }`), &obj.Object); err != nil { t.Fatalf("error decoding YAML: %v", err) </del> <ins> "data": { "key": "value" } }`)) if err != nil { t.Fatal(err) </ins> } <ins> previousManagedFields := f.ManagedFields() time.Sleep(time.Second) </ins> <del> if err := f.Update(obj, "fieldmanager_test"); err != nil { t.Fatalf("failed to update object: %v", err) </del> <ins> err = updateObject(&f, "fieldmanager_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key": "new-value" } }`)) if err != nil { t.Fatal(err) </ins> } <del> managed := f.ManagedFields() obj2 := &unstructured.Unstructured{Object: map[string]interface{}{}} if err := yaml.Unmarshal([]byte(`{ </del> <ins> newManagedFields := f.ManagedFields() if previousManagedFields[0].Time.Equal(newManagedFields[0].Time) { t.Errorf("ManagedFields time has not been updated:n%v", newManagedFields) } } func TestManagedFieldsApplyDoesModifyTime(t *testing.T) { var err error f := NewTestFieldManager(schema.FromAPIVersionAndKind("v1", "ConfigMap"), "", nil) err = applyObject(&f, "fieldmanager_test", []byte(`{ </ins> "apiVersion": "v1", <del> "kind": "Pod", </del> <ins> "kind": "ConfigMap", </ins> "metadata": { <del> "name": "pod", "labels": {"app": "nginx2"} </del> <ins> "name": "configmap" </ins> }, <del> }`), &obj2.Object); err != nil { t.Fatalf("error decoding YAML: %v", err) </del> <ins> "data": { "key": "value" } }`)) if err != nil { t.Fatal(err) } previousManagedFields := f.ManagedFields() time.Sleep(time.Second) err = applyObject(&f, "fieldmanager_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key": "new-value" } }`)) if err != nil { t.Fatal(err) } newManagedFields := f.ManagedFields() if previousManagedFields[0].Time.Equal(newManagedFields[0].Time) { t.Errorf("ManagedFields time has not been updated:n%v", newManagedFields) } } func TestManagedFieldsUpdateWithoutChangesDoesNotModifyTime(t *testing.T) { var err error f := NewTestFieldManager(schema.FromAPIVersionAndKind("v1", "ConfigMap"), "", nil) err = updateObject(&f, "fieldmanager_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key": "value" } }`)) if err != nil { t.Fatal(err) } previousManagedFields := f.ManagedFields() time.Sleep(time.Second) err = updateObject(&f, "fieldmanager_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key": "value" } }`)) if err != nil { t.Fatal(err) } newManagedFields := f.ManagedFields() if !previousManagedFields[0].Time.Equal(newManagedFields[0].Time) { t.Errorf("ManagedFields time has changed:nBefore:n%vnAfter:n%v", previousManagedFields, newManagedFields) } } func TestManagedFieldsApplyWithoutChangesDoesNotModifyTime(t *testing.T) { var err error f := NewTestFieldManager(schema.FromAPIVersionAndKind("v1", "ConfigMap"), "", nil) err = applyObject(&f, "fieldmanager_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key": "value" } }`)) if err != nil { t.Fatal(err) } previousManagedFields := f.ManagedFields() time.Sleep(time.Second) err = applyObject(&f, "fieldmanager_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key": "value" } }`)) if err != nil { t.Fatal(err) } newManagedFields := f.ManagedFields() if !previousManagedFields[0].Time.Equal(newManagedFields[0].Time) { t.Errorf("ManagedFields time has changed:nBefore:n%vnAfter:n%v", previousManagedFields, newManagedFields) } } func TestNonManagedFieldsUpdateDoesNotModifyTime(t *testing.T) { var err error f := NewTestFieldManager(schema.FromAPIVersionAndKind("v1", "ConfigMap"), "", nil) err = updateObject(&f, "fieldmanager_a_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key_a": "value" } }`)) if err != nil { t.Fatal(err) } err = updateObject(&f, "fieldmanager_b_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key_b": "value" } }`)) if err != nil { t.Fatal(err) </ins> } <ins> previousManagedFields := f.ManagedFields() previousEntries := map[string]v1.ManagedFieldsEntry{} for _, entry := range previousManagedFields { previousEntries[entry.Manager] = entry } time.Sleep(time.Second) err = updateObject(&f, "fieldmanager_a_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key_a": "value", "key_b": "new-value" } }`)) if err != nil { t.Fatal(err) } newManagedFields := f.ManagedFields() newEntries := map[string]v1.ManagedFieldsEntry{} for _, entry := range newManagedFields { newEntries[entry.Manager] = entry } if _, ok := newEntries["fieldmanager_b_test"]; ok { t.Errorf("FieldManager B ManagedFields has changed:n%v", newEntries["fieldmanager_b_test"]) } } func TestNonManagedFieldsApplyDoesNotModifyTime(t *testing.T) { var err error f := NewTestFieldManager(schema.FromAPIVersionAndKind("v1", "ConfigMap"), "", nil) err = applyObject(&f, "fieldmanager_a_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key_a": "value" } }`)) if err != nil { t.Fatal(err) } err = applyObject(&f, "fieldmanager_b_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key_b": "value" } }`)) if err != nil { t.Fatal(err) } previousManagedFields := f.ManagedFields() previousEntries := map[string]v1.ManagedFieldsEntry{} for _, entry := range previousManagedFields { previousEntries[entry.Manager] = entry } time.Sleep(time.Second) err = applyObject(&f, "fieldmanager_a_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key_a": "new-value" } }`)) if err != nil { t.Fatal(err) } newManagedFields := f.ManagedFields() newEntries := map[string]v1.ManagedFieldsEntry{} for _, entry := range newManagedFields { newEntries[entry.Manager] = entry } if !previousEntries["fieldmanager_b_test"].Time.Equal(newEntries["fieldmanager_b_test"].Time) { t.Errorf("FieldManager B ManagedFields time changed:nBefore:n%vnAfter:n%v", previousEntries["fieldmanager_b_test"], newEntries["fieldmanager_b_test"]) } } func TestTakingOverManagedFieldsDuringUpdateDoesNotModifyPreviousManagerTime(t *testing.T) { var err error f := NewTestFieldManager(schema.FromAPIVersionAndKind("v1", "ConfigMap"), "", nil) err = updateObject(&f, "fieldmanager_a_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key_a": "value", "key_b": value" } }`)) if err != nil { t.Fatal(err) } previousManagedFields := f.ManagedFields() previousEntries := map[string]v1.ManagedFieldsEntry{} for _, entry := range previousManagedFields { previousEntries[entry.Manager] = entry } time.Sleep(time.Second) err = updateObject(&f, "fieldmanager_b_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key_b": "new-value" } }`)) if err != nil { t.Fatal(err) } newManagedFields := f.ManagedFields() newEntries := map[string]v1.ManagedFieldsEntry{} for _, entry := range newManagedFields { newEntries[entry.Manager] = entry } if !previousEntries["fieldmanager_a_test"].Time.Equal(newEntries["fieldmanager_a_test"].Time) { t.Errorf("FieldManager A ManagedFields time has been updated:nBefore:n%vnAfter:n%v", previousEntries["fieldmanager_a_test"], newEntries["fieldmanager_a_test"]) } } func TestTakingOverManagedFieldsDuringApplyDoesNotModifyPreviousManagerTime(t *testing.T) { var err error f := NewTestFieldManager(schema.FromAPIVersionAndKind("v1", "ConfigMap"), "", nil) err = applyObject(&f, "fieldmanager_a_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key_a": "value", "key_b": value" } }`)) if err != nil { t.Fatal(err) } previousManagedFields := f.ManagedFields() previousEntries := map[string]v1.ManagedFieldsEntry{} for _, entry := range previousManagedFields { previousEntries[entry.Manager] = entry } </ins> time.Sleep(time.Second) <del> if err := f.Update(obj2, "fieldmanager_test"); err != nil { t.Fatalf("failed to update object: %v", err) </del> <ins> err = applyObject(&f, "fieldmanager_b_test", []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "configmap" }, "data": { "key_b": "new-value" } }`)) if err != nil { t.Fatal(err) </ins> } <del> if !reflect.DeepEqual(managed, f.ManagedFields()) { t.Errorf("ManagedFields changed:nBefore:n%vnAfter:n%v", managed, f.ManagedFields()) </del> <ins> newManagedFields := f.ManagedFields() newEntries := map[string]v1.ManagedFieldsEntry{} for _, entry := range newManagedFields { newEntries[entry.Manager] = entry } if !previousEntries["fieldmanager_a_test"].Time.Equal(newEntries["fieldmanager_a_test"].Time) { t.Errorf("FieldManager A ManagedFields time has been updated:nBefore:n%vnAfter:n%v", previousEntries["fieldmanager_a_test"], newEntries["fieldmanager_a_test"]) </ins> } }

I created an object with , and later I updated the same object with another . After that I looked at the object's managedFields, and saw nothing about the update. Here is what I got after the second apply: I expected managedFields to get updated by the second . No response <details</details<detailsnone </details<details</details<detailskubeadm </details<detailsdocker </details<details</details
/cc
I also saw the timestamp not updated when I updated using client-go and typed client stubs, Apply method, patch type .
/assign /triage accepted
/cc
The is not updated because you are not changing the structure of the object (adding or removing fields). According to , the time is updated if and only if the fieldSet is changed. It does not take into account the value of fields. One option would be to not compare the fieldSets but directly compare the liveObject & newObject for differences and update the field accordingly. Is it the desired behavior to update the time of when only the value is updated?
Yes! Look at the first paragraph of .
is that documentation correct? Is this what we intended?
I don't see any mention to the timestamp in that first paragraph. We've clarified recently the exact semantic of the timestamp, as defined here: I think you've identified the wrong code if you want to go ahead and fix it, go ahead, I'll happily review!


Question: crdClient client.CustomResourceDefinitionsGetter, ) *ConditionController { c := &ConditionController{ <del> crdClient: crdClient, crdLister: crdInformer.Lister(), crdSynced: crdInformer.Informer().HasSynced, queue:     workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "non_structural_schema_condition_controller"), </del> <ins> crdClient:          crdClient, crdLister:          crdInformer.Lister(), crdSynced:          crdInformer.Informer().HasSynced, queue:              workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "non_structural_schema_condition_controller"), lastSeenGeneration: map[string]int64{}, </ins> } crdInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc:    c.addCustomResourceDefinition, UpdateFunc: c.updateCustomResourceDefinition, <del> DeleteFunc: nil, </del> <ins> DeleteFunc: c.deleteCustomResourceDefinition, </ins> }) c.syncFn = c.sync

/sig api-machinery /kind bug /assign sttts because the controller that maintains the condition runs multiple instances in HA apiserver clusters, the controller must not duel in cases where the validation messages change slightly between releases


Question: "//pkg/apis/componentconfig:go_default_library", "//pkg/kubelet/api:go_default_library", "//pkg/kubelet/api/v1alpha1/runtime:go_default_library", <ins> "//pkg/kubelet/cm:go_default_library", </ins> "//pkg/kubelet/container:go_default_library", "//pkg/kubelet/dockershim/cm:go_default_library", "//pkg/kubelet/dockertools:go_default_library",

Kubernetes version (use ): v1.5.0-alpha.1.2189+-dirty docker: 1.12.3 Environment: fedora24 docker-engine-1.12.3-1.fc24.x86_64 What happened: if docker Cgroup Driver=cgroupfs, and set --cgroup-driver=systemd in kubelet, then enable --experimental-cgroups-per-qos=true; The container cgroup some is not correct. How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: 1) docker Cgroup Driver=cgroupfs and --cgroup-driver=cgroupfs in kubelet works well 2) docker Cgroup Driver=systemd and --cgroup-driver=systemd in kubelet works well
help check, thanks.
In Kubernetes 1.6, we can work to detect this case and return an error. IMHO its a configuration error to have docker running with a cgroupfs driver paired with the kubelet running the systemd cgroup driver. /cc
We can also document the compatibility matrix for now.


Question: podState, ok := cache.podStates[key] if !ok { <del> return nil, fmt.Errorf("pod %v does not exist", key) </del> <ins> return nil, fmt.Errorf("pod %v does not exist in scheduler cache", key) </ins> } return podState.pod, nil

/kind bug There are several error messages in the scheduler cache that are confusing to read, at least one of these surfaced up to end users via Events. I noticed these two in particular but there may be others: Whether or not these should actually be surfacing as events, the messages should be cleaned up.
cc
Seems that these are internal or implement details in scheduler, do our end users can fully understand what the events really mean? I think it is depend on our own implement. idea?
I believe, the problem is with in the message but I am not able to reproduce it, can you please share steps to reproduce this.
No unfortunately, one of our users reported seeing it when there system was in a bad state, not in a reproducible way. The point of the issue is just that the grammar in those messages is bad, which should be fixable even if you can't get them to fire.


Question: framework.ExpectNoError(verifyServeHostnameServiceUp(cs, ns, podNames3, svc3IP, servicePort)) }) <ins> ginkgo.It("should work after the service has been recreated", func() { serviceName := "service-deletion" ns := f.Namespace.Name numPods, servicePort := 1, defaultServeHostnameServicePort ginkgo.By("creating the service " + serviceName + " in namespace " + ns) defer func() { framework.ExpectNoError(StopServeHostnameService(f.ClientSet, ns, serviceName)) }() podNames, svcIP, _ := StartServeHostnameService(cs, getServeHostnameService(serviceName), ns, numPods) framework.ExpectNoError(verifyServeHostnameServiceUp(cs, ns, podNames, svcIP, servicePort)) ginkgo.By("deleting the service " + serviceName + " in namespace " + ns) err := cs.CoreV1().Services(ns).Delete(context.TODO(), serviceName, metav1.DeleteOptions{}) framework.ExpectNoError(err) ginkgo.By("Waiting for the service " + serviceName + " in namespace " + ns + " to disappear") if pollErr := wait.PollImmediate(framework.Poll, e2eservice.RespondingTimeout, func() (bool, error) { _, err := cs.CoreV1().Services(ns).Get(context.TODO(), serviceName, metav1.GetOptions{}) if err != nil { if apierrors.IsNotFound(err) { framework.Logf("Service %s/%s is gone.", ns, serviceName) return true, nil } return false, err } framework.Logf("Service %s/%s still exists", ns, serviceName) return false, nil }); pollErr != nil { framework.Failf("Failed to wait for service to disappear: %v", pollErr) } ginkgo.By("recreating the service " + serviceName + " in namespace " + ns) svc, err := cs.CoreV1().Services(ns).Create(context.TODO(), getServeHostnameService(serviceName), metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(verifyServeHostnameServiceUp(cs, ns, podNames, svc.Spec.ClusterIP, servicePort)) }) </ins> ginkgo.It("should work after restarting kube-proxy [Disruptive]", func() { kubeProxyLabelSet := map[string]string{clusterAddonLabelKey: kubeProxyLabelName} e2eskipper.SkipUnlessComponentRunsAsPodsAndClientCanDeleteThem(kubeProxyLabelName, cs, metav1.NamespaceSystem, kubeProxyLabelSet)

Creating a destroying services test is a required test case in , but it's missing in the upstream e2e test. Since creating a destroying services test should work in both windows and linux cluster, it's worthy adding it to create a service and corresponding pod delete the service re-create the service load balancing rules for pods are recovered e2e tests for sig-network would have coverage for recover load balancing rules for pods after service is deleted and re-created. Reproducer is above No response <detailsmain </details<detailsany </details<details</details<details</details<details</details<details</details
/sig network testing
/triage accepted /assign


Question: } func (az *Cloud) processHTTPResponse(service *v1.Service, reason string, resp *http.Response, err error) error { <del> if isSuccessHTTPResponse(resp) { </del> <ins> if err == nil && isSuccessHTTPResponse(resp) { </ins> // HTTP 2xx suggests a successful response return nil }

What happened: In a StatefulSet where each Pod has its own persistent volume provisioned by , it occasionally takes up to 30 min to detach and re-attach those disks to their node during a rollout. The following Pod events repeat: A subset of the controller-manager's logs (full logs on demand): State of the disk few minutes before the attach operation succeeds, returned by the Azure API: In this particular VMSS, each instance has at most 2 disks attached. In busy clusters, a node can have up to 5 disks attached. The operation eventually succeeds, ~20m later in this case: What you expected to happen: Disks take no longer than 3 min to re-attach (the typical time it takes in our customers' environments). How to reproduce it (as minimally and precisely as possible): Here is a copy of the manifest of the affected StatefulSet: <details<summarysolr-</summary</details</brStorage class <details<summarydefault-</summary</details</brAnything else we need to know?: Environment: Kubernetes version: v1.12.7 Cloud provider: Azure (via AKS-Engine) OS (e.g: ): Ubuntu 16.04.6 LTS Kernel: 4.15.0-1042-azure Install tools: Network plugin and version: Others: observed in westeurope and eastus regions paging /sig azure
can you also provide following info on master node: How many disks are on one VM? We are experiencing slow disk attach/detach issue when disk num is large:
I just the info about disks to the original message: between 2 and 5 per VMSS instance. The command you posted returns nothing, but if I remove "vm" from the last grep I get: Feel free to catch up on Slack if you want to have a real-time chat.
/assign paste some info here: the 2 disk attach/detach operations costs s totally, that's reason why it's stuck in so long time. BTW, there is metrics calculation issue here: , but anyway the two operations cost too long time in vmss API call level.
this seems to be an extreme case. In other environments where the issue occurs, API calls cost between 15 and 60 sec on average.

API calls cost between 15 and 60 sec on average and still takes up 30min to re-attach? could it repro easily?
Only in selected clusters unfortunately (about 5%), but when it happens it happens quite consistently. I haven't been able to reproduce outside of clusters belonging to live customers, which makes the issue a bit tricky to troubleshoot at the moment. I'm sitting in front of a cluster where it's currently happening. A Pod has been reporting for 53 min. Yesterday I observed it in the same cluster for 2 hours. We eventually deleted the Pod to unblock the situation (which doesn't always work btw). The following Pod events repeat over and over again: The disk has and just like in my first message. The controller-manager metrics also don't tell us much (14h uptime):
could you provide detailed steps about how to repro this issue? I tried to drain statefulset pods from one node to another on k8s v1.12.7, cannot repro.
The correct way to get disk attach/detach metrics on k8s v1.12.7 is like this The rolling update would cause k8s to detach disk first and then attach disk after, so I will still get following error in my testing env, but finally it could recover:
This one is a patched 1.10, with your retry fix. I have some metrics from an affected 1.12 cluster, but like I said, nothing very useful: We only have one customer on 1.12 so I'm waiting for the next incident to get back to you on a proper 1.12 setup. My understanding of what happens in most (if not all) cases is: Imagine this simplified setup 2 nodes 3 Pods belonging to a Deployment "commerce" 3 Pods belonging to a Deployment "api" 3 Pods with Azure Disks belonging to a StatefulSet "solr" We update both Deployments and the StatefulSet simultaneously (we have an operator that actually does exactly this). Both and terminate before . This time and are both scheduled on . is still Terminating. When is finally down, is already packed, so the scheduler has no other choice but to move it to . And then we are in that mode where the disk is detached, but never re-attached.

It shouldn't be any different from a drain, but at least it can maybe help you understand our typical repro scenario. We recently optimized scheduling to avoid that schenario where stateful Pods must be relocated, but other events can cause relocation like nodes failures, autoscaling, ...
here is the conclusion about what we found now: Periodically k8s cluster failed to attach a disk to a VMSS instance, that disk is actually in “unattached” state, it's caused by vmss instance in unhealthy state, although by azure cli command , the instance shows healthy. workaround: run to update the VMSS instance manually, it could recover in a minute or so We are asking VMSS team to find the root cause of this issue. In brief, if you find k8s could not attach an state disk to a VMSS node, it's probably due to the vmss instance issue, run to update the VMSS instance manually could solve this issue.


Question: claimWithClass("silver", newClaimArray("claim11-13", "uid11-13", "1Gi", "", v1.ClaimPending)), // Binding will be completed in the next syncClaim claimWithClass("silver", newClaimArray("claim11-13", "uid11-13", "1Gi", "", v1.ClaimPending, annStorageProvisioner)), <del> noevents, noerrors, wrapTestWithProvisionCalls([]provisionCall{provision2Success}, testSyncClaim), </del> <ins> []string{"Normal ProvisioningSucceeded"}, noerrors, wrapTestWithProvisionCalls([]provisionCall{provision2Success}, testSyncClaim), </ins> }, { // Provision error - non existing class

is quite confusing when provisioning for a PVC fails and then succeeds: it shows the PVC as , i.e. everything is OK the last logged event is , i.e. something went wrong What of the above is correct? We should log also the success:
fixed by


Question: <ins> /* Copyright 2016 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // Package volume implements a controller to manage volume attach and detach // operations. package volume </ins>

e2e-cos-alpha-features All I'm not sure it was ever running. No response No response /sig node /priority important-longterm since it's alpha features, setting priority lower
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
kubelet log shows error below:
We tried to fix it in before. I opened


Question: Force:           force, ServerSideApply: o.ServerSideApply, ForceConflicts:  o.ForceConflicts, <ins> IOStreams:       o.Diff.IOStreams, </ins> } err = differ.Diff(obj, printer)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I tried to with a Deployment spec that was roughly shaped like this (note that the is invalid, since it should be a list of instead of ). This caused kubectl to fail to create a three-way patch and panic. <details<summaryClick here for the stacktrace</summary</detailsWhat you expected to happen: An error is returned How to reproduce it (as minimally and precisely as possible): Create a deployment, then try to diff using the YAML provided above (make sure to replace the name and namespace) Anything else we need to know?: The panic occurs on this line because is nil: It's set to nil here: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig cli
Raised a pr which may fix this bug.


Question: } totalNumNodes := len(nodeInfos) <del> score := calculatePriority(sumImageScores(nodeInfo, pod.Spec.Containers, totalNumNodes)) </del> <ins> score := calculatePriority(sumImageScores(nodeInfo, pod.Spec.Containers, totalNumNodes), len(pod.Spec.Containers)) </ins> return score, nil }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : redefine the formula, the score should be capped per images instead of pod. Why is this needed: For the pod that going to be scheduled, calculate the score based on the below steps, Scaled score base on the image size and it's spread over the cluster, image[i].nodes: number of nodes that already have image ! Capped score based on the assumption 90% image size on dockerhub falls into the range [23M ~ 1000M] lowerbound: 23M upperbound:1000M ! Final scores returned for scheduling, maxscore: 100 ! The second step is not well guarded, the assumption is based on the image size while the pod may hold multiple images, it is not surprise to see that total image size of the pods is larger than 1000M, while each of might be only around 300M. In this below case, following the above formula, NODE1 and NODE2 willl get the same scores(100), since both of them are capped by the max score (100), the ideal result is score of NODE2 should be larger than NODE1 (all requested images are ready on NODE2) and is preferred, the larger the sample1 here the more scores should be expected for NODE2. !
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig scheduler Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
cc
Thanks a lot for summarizing the problem. I agree that the formula needs improvement. ~I think doing the capping for each image solves the original problem and what said in There are two options: only the total from to for each container. We have to ask ourselves how much signal each formula gives and depending on what. Think about different scenarios: a pod with a lot of very small containers, a pod with a few big containers, etc.
/assign
Here is my thoughts, step1: calculate image level priority, for each image (should be same as exsting func ) step2: cap the each image's score with thresholds, minThreshold and maxThreshold.

(should be same as existing func except that it's per image instead of pod) step3: score normalization per image. step4: calculate pod level priority, and return the final score, range (0~100) step1 and step2 primarily update the exiting code to make it is image level priorities instead of pod level. Based on the above steps, consider the below two cases, a pod with four images, case1: All images are ready on node2 node1 (0, 80, 90, 60) - final score is: (80+90+60) / 4 = 57 node2 (90, 80, 90, 60) - final score is: (90+80+90+60) / 4 = 80 √ case2: each node has three images ready but image1 is larger than image2. node1 (0, 80, 70, 60) - final score is: (80+70+60) / 4 = 52 node2 (90, 0, 70, 60) - final score is: (90 +70+60) / 4 = 55 √ thoughts? cc
Thanks can you update the the examples with the score that we would have gotten with the current function?
For reference, see this analysis for a different plugin
Also it's worth to see the difference with just capping per pod, instead of per container.
can you also describe the image sizes you used? the example in the original post assumed three images, while your analysis is assuming four.
agree with and the explanation in looks good but just to be sure it would be nice to see the math worked out with image sizes, and compared to the current formula
Sorry for the confusion, let me switch back to the original three images, update the image size a little bit to make it easier to understand, and also fit to the case of and . Here is the pod, ! and three nodes in the cluster, ! Let's see the difference between two approaches, capping per pod, step1: ! NODE1: scores = 30 2/3 + 1500 2/3 = 1020 NODE2: scores = 30 2/3 + 600 2/3 + 1500 2/3 = 1420 step2: !

NODE1: 1020 1000 (maxThreshold), cappedscore = 1000 NODE2: 1420 1000 (maxThreshold), cappedscore = 1000 step3: ! upperbound = 100 / lowerbound = 23 / maxscore = 100 NOTE1: score = 100 (1000 - 23) / (1000 - 23) = 100 NODE2: score = 100 (1000 - 23) / (1000 - 23) = 100 potentially, NODE1 will be chose for scheduling. capping per images, step1: calculate image level priority, NODE1: image1: score = 30 2/3 = 20 image2: score = 0 image3: score = 1500 2/3 = 1000 NODE2: image1: score = 30 2/3 = 20 image2: score = 600 2/3 = 400 image3: score = 1500 2/3 = 1000 step2/3: cap each image's score with thresholds and calculate the image score, ! NODE1: image1: score = 100 (23 - 23) / ( 1000 - 23) = 0 image2: score = 0 image3: score = 100 (1000 - 23) / (1000 - 23) = 100 NODE2: image1: score = 100 (23 - 23) / ( 1000 - 23) = 0 image2: score = 100 (400 - 23) / (1000 - 23) = 35 image3: score = 100 * (1000 - 23) / (1000 - 23) = 100 step4: final score NODE1: finalscore = 100 / 3 = 33 NODE2: final_score = 135 / 3 = 45 NODE2 will be chosen for scheduling.
Computing the score per image and normalizing by the number of images make sense to me at first glance, but I would like to see more examples.

It would be much easier to digest the analysis if it was summarized in tabular form comparing three scoring functions: (1) current one, (2) one in and (3) multiple big images, multiple small images (total less than the threshold) and a mix. Also, compare cases where the spread is high vs low.
Yes, we don't need to see the individual steps, but a good set of scenarios with final scores. I didn't mean just the old function. We know it's very ineffective. What I meant is capping per pod using the strategy you used in
Did some comparison here, spread has been included based on the layout of each node, it's not easy for me to pull this factor out to focus on this factor only. I will design more testcases when the patch is ready for review which I hope could be more clear. column 1: pod with three images column 2: three nodes' images layout column 3 / 4 / 5: score of each node for different approaches. pod cap and normalizing per image
[10, 20, 30] & small [0,0,0] [800, 1200, 1500] & big [33, 59, 67] [30, 600, 1500] & mix [33, 45, 0]
Normalizing by the number of images gives each image the same "weight". Another variation of the third option is to try and multiply each image score with its size divided by the total size of all images, can you add that as another option?
I don't understand that suggestion. The score of an image is already proportional to its size. I would say the difference between the two suggestions is: Capping per pod cares about total downloaded size Capping per image gives roughly the same importance to each image. I think capping per pod makes more sense. However, there is some information we are losing in both suggestions, as seen in the first case. Even if a lot of small images need to be downloaded, we are giving a 0 score to all nodes. I think a cap from makes more sense, as suggested in the initial PR.
It helps giving more importance to larger images.
If the statistics are accurate with regards to image size distribution, then having many small images all scoring zero isn't common.

For the more common cases (sizes between 32-1000), it seems that both approaches will likely give similar results, so I am fine with both approaches.
That would make the score proportional to the square of the size. Just to play safe (ie, not make the score too strong in a way that could affect spreading or resource allocation) I suggest we go with capping per pod. And then we can re-evaluate next release.
Speaking of statistics. They might have changed since this plugin was first developed. Is there a reliable source?
True for uncapped images, but everything we are doing is just heuristics, and the suggestion is just for the sake of exploring options, not advocating a specific one. Playing it safe sounds good to me.
This approach needs the smallest change and the minimum impact.
+1, this is not common case we can see that a pod with lots of small images (all < 23M), but a corner case that might be improved. No perfect solution so far, but either one is better than the original one IMHO.
/cc , Do you have any comments so far?
btw, has been updated to the second approach (capping and normalization per image).
I would go with this approach as well, it preserves the behavior for multiple small images, and improve it for multiple big ones.
seems like a simple enough compromise for now, and running through some of the examples with that formula it does get more of the specificity you're looking for with large images without normalizing all smaller images right to 0 (so we keep some of the detail there too). I would like to see some updated statistics on the thresholds, and while 90% of the images might fall between them I'm wondering if that's a totally normal distribution. suggestion in got me (tangentially) thinking if we could use thresholds that are more relevant to the pod in question, rather than all of Dockerhub. Because as it is now hypothetical pods (and clusters) which have all small images don't get much benefit from ImageLocality. But I recognize that could be intentional since download resources for small images is ultimately negligible.
I also believe that scoring of small images by zero is intentional, but seems like making lots of small images in pod accountable for the scheduling is slightly better.
It looks like we have consensus. Can you update the PR?


Question: "fmt" "io" "net" <ins> "sort" </ins> "strconv" "strings" "time"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature isn't very useful as it only shows the name and age of the resourcequota resource. It'd be nice if it also shows the current/allocated usage for each, like this: Each resourcequota summary is of the format: . I think it would be a useful feature. I wouldn't mind taking a crack at implementing it if it's deemed worth by the maintainers. Thanks.
/sig cli
/assign code-sleuth
GitHub didn't allow me to assign the following users: code-sleuth. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign code-sleuth Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign juanvallejo