Question: if err != nil { return true, err } <del> rootStat, err := os.Lstat(filepath.Dir(strings.TrimSuffix(file, "/"))) </del> <ins> rootStat, err := os.Stat(filepath.Dir(strings.TrimSuffix(file, "/"))) </ins> if err != nil { return true, err }

What happened: In our kubernetes cluster, there are some slave nodes whose root partition has a very little disk space, so we add another larger-size disk device and symbolically link to a directory located at this larger device. however, kubelet failed to delete orphaned pod directory, there are lots of spam logging as below: After diving into this problem, I found the root cause: the failed to distinguishing the mountpoint when is a symbolic link. just compare the device number of the pod directory and its parent directory. however, using to obtain parent directory's device number which doesn't follow the link can lead an incorrect result I have done some tests: What you expected to happen: kubelet can successfully delete orphaned pod directory when the kubelet's pods directory symbolically links to another device's directory How to reproduce it (as minimally and precisely as possible): link to another disk device's directory pods and delete them Anything else we need to know?: Environment: Kubernetes version (use ): v1.10.11 Cloud provider or hardware configuration: bare-metal cluster OS (e.g: ):Kernel (e.g. ): Linux sh2-com-ocean120- 4.9.2-3.0.0.std7b.el7.5.x8664 SMP Thu Apr 26 17:33:02 CST 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /sig storage
me too


Question: # - 1: fatal error - cluster won't be working correctly # - 2: weak error - something went wrong, but cluster probably will be working correctly # We just print an error message in case 2). <del> if [[ "${validate_result}" == "1" ]]; then </del> <ins> if [[ "${validate_result}" -eq 1 ]]; then </ins> exit 1 <del> elif [[ "${validate_result}" == "2" ]]; then </del> <ins> elif [[ "${validate_result}" -eq 2 ]]; then </ins> echo "...ignoring non-fatal errors in validate-cluster" >&2 fi fi

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): upgrade test leak Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): 1.5 beta Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: kube- What happened: While doing manual upgrade testing, found that I couldn't start a new cluster as the existing firewall rules and external master ip address reservation where preventing kube-up from succeeding after running kube-down What you expected to happen: kube-down to clean up all cluster related resources How to reproduce it (as minimally and precisely as possible):
I tried this on a different GCP project and it didn't reproduce. I'm going to see if there's anything strange about my project.
Ack marking this as non-release-blocker
I think I found the problem. (from ) evaluates to false in bash on my mac. Changing it to fixes my issue. I'll send out a PR.


Question: imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo" <del> "github.com/onsi/gomega" </del> ) const (

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /priority critical-urgent /cc /milestone v1.19 This test waits for the limit range to be deleted before attempting to create the so this error is surprising. It appears there are no limit ranges present when they are listed here:
It seems this job failed just once since Jul 23, so I'm wondering is it a must for 1.19?
Given this is a conformance test I would prefer we at least know what is causing the issue
This is still flaking with this same issue. /reopen
Reopened this issue. <detailsIn response to : This is still flaking with this same issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.20
This issue has not occurred recently and seems to have already been fixed. I think we can close this issue.
Yes, was expected to get it fixed. /close
Closing this issue. <detailsIn response to : Yes, was expected to get it fixed. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this is still flaking in the 1.20 branch : confused : as of one hour ago


Question: otelhttp.WithPropagators(tracing.Propagators()), otelhttp.WithPublicEndpoint(), otelhttp.WithTracerProvider(tp), <ins> otelhttp.WithSpanNameFormatter(func(operation string, r *http.Request) string { ctx := r.Context() info, exist := request.RequestInfoFrom(ctx) if !exist || !info.IsResourceRequest { return r.Method } return getSpanNameFromRequestInfo(info, r) }), </ins> } wrappedHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Add the http.target attribute to the otelhttp span

When enable apiserver tracing, the webhook request spans only show spanName as /, mostly because use http request operations as default span name, which is hard to understand. ! Following the open-telemetry standard of spanName, it's recommended to use as spanName. ! SpanName of webhook request should be , like ! APIServerTracing feature gates of apiserver a pod tracing dashboard No response <details18 </details<detailsNone </details<details</details<details</details<details</details<details</details
I make a fix for this.
/sig api-machinery instrumentation
Do you know if this is fixed in later versions of opentelemetry?
/triage accepted /assign
Looks like might be related
not yet, I think we can fix this in otel upstream, but it's hard to backport to otel , which is the version of otel used in Kubernetes
Good point. It think fixing it here is a good idea in that case.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
could I take on this issue?
yes. Feel free to work on this.
! I test locally by 1.29 kubernetes and start a sample webhook, looks like the client span has already conforms to semantic conventions , which should be in the form of "HTTP {METHODNAME}"
! For apiserver as a server, now the spanName is still KubernetesAPI, which I think we could enhance it into "HTTP get_account/{accountId}" format
After digging for a while, I found one solution to implement this, I draft this idea in , could you help review about whether this is a correct way to implement it?


Question: return "", fmt.Errorf("created security group, but id was not returned: %s", name) } <del> tags := []*ec2.Tag{} for k, v := range s.filterTags { tag := &ec2.Tag{} tag.Key = aws.String(k) tag.Value = aws.String(v) tags = append(tags, tag) } if len(tags) > 0 { tagRequest := &ec2.CreateTagsInput{} tagRequest.Resources = []*string{&groupID} tagRequest.Tags = tags if _, err := s.createTags(tagRequest); err != nil { // Not clear how to recover fully from this; we're OK because we don't match on tags, but that is a little odd return "", fmt.Errorf("error tagging security group: %v", err) } </del> <ins> err := s.createTags(groupID, s.filterTags) if err != nil { // If we retry, ensureClusterTags will recover from this - it // will add the missing tags.  We could delete the security // group here, but that doesn't feel like the right thing, as // the caller is likely to retry the create return "", fmt.Errorf("error tagging security group: %v", err) </ins> } return groupID, nil }

After setting up a few services, I noticed the following output in : I assume the service tried to configure an ELB in the past (not sure when or how or what details it tried to use) and has kept trying and now can't even use the AWS API. How can I either: 1) tell it to please stop so that I can use my AWS account again 2) fix it so that it does configure the ELB /cc
Sorry about the problem. Looks like we need a sleep / backoff policy here! To get you up and running again, if you redeploy the nginx service without Type=LoadBalancer, we will (should) stop trying to create a load balancer, and you should get out of the AWS API penalty-box. Sorry about this, and thanks for the report. Looks like this is pretty important to fix. Do you happen to know why you can't create an ELB? My guess is that you have hit your quota?
I'll remove the service, wait for the penalty-box to expire, then build it again so that we can see what the real error was.
Adding the LoadBalancer service back in, I'm seeing:
I took a few different steps and seem to have gotten it both working in different ways: First I deleted the service because it was erroring, but I noticed the count was still going up. So I deleted the security group it said was a duplicate and then it finished setting the ELB up! That meant I needed to manually clean up the ELB resources and security groups because k8 thought it was already deleted. After cleaning up all the resources in AWS again, I the service again and it seemed to work fine! I'm not sure how/why it initially confused itself with the duplicate SG, but it doesn't necessarily seem to happen every time. The code could likely just, in the event that it sees the SG already exists, continue on with the setup. I could send you a PR to handle that if you'd like!
I have a PR that I think covers this: . I thought it was a one-off error, but it sounds like it wasn't. Thanks so much for pausing & retrying also - that was super-helpful. Even if we think covers this, let's leave this issue open to track the need for a backoff to avoid overloading the AWS API.

(And presumably this aggressive retrying happens on other clouds also, though maybe they have overload a sleep built in).
Certainly looks like should fix it. I'm still wondering why I'd see it failing literally for hours in a loop though. If it was a problem with AWS being eventually consistent, then after an hour I'd think that first check, looking to see if the securityGroup existed, would start to return the group! I don't know the k8 code well yet, though so perhaps it was looping in a different place that wasn't running that code to see if the security group already existed.
you're right; my explanation wasn't entirely correct... So security group names are unique per VPC. We also have a per-cluster name, and we filter all our AWS API interactions to only select resources tagged with this name. If we had a security group name collision across two k8s clusters in the same VPC, that would be bad. Thankfully the security group name we're creating includes a UUID, so this "won't happen". However, the problem is that we created the security group, and then failed to tag it. But we were filtering based on our cluster. So our "describe security groups" operation would never see the SG, because it wasn't appropriately tagged. We have a few workarounds to make this much less likely, and I have opened the "big picture" issue . And I'm going to think about whether we should in fact be filtering when we DescribeSecurityGroups - I think the answer is no in the case where the name has a UUID in it.
Ah! That totally makes sense. The race between resource creation and resource tagging can created "invisible" resources if the process of tagging fails. Maybe an anti-entropy function that can look at all the resources and normalize them. In the case of invisible resources, you can just delete them and hope that recreating them won't fail the tagging process.
I see this as well when frequently deploying/testing. Unfortunately, it happens far too often (running v1.0.0).
+1
I've been having a similar issue to this, Kubernetes can't create a load balancer as it says the security group already exists. The workaround I've found is to manually delete the security group and wait for Kubernetes to successfully create it again.

I've generally had the problem when quickly deleting a service with a and then creating it again (I see now I should use ). I wonder if it is something to do with AWS reassigning the same load balancer back to you if you haven't finished deleting it coupled with the tagging issue mentioned earlier.
+1
cc
I had an email discussion with on this and he said that what's remaining shouldn't be considered a blocker for 1.1, so I'll move it to 1.2-candidate.
Can you summarize what's remaining to do for this issue? Should we just close it in favor of ?
I see you the v1.2 milestone -- will you be able to look into this in the next few days?


Question: if podAddErr != nil { t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podAddErr) } <del> waitForVolumeAttachedToNode(t, generatedVolumeName, nodeName2, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName2, cache.AttachStateAttached, asw) </del> <ins> waitForVolumeAttachStateToNode(t, generatedVolumeName, nodeName2, cache.AttachStateAttached, asw) </ins> } // Populates desiredStateOfWorld cache with one node/volume/pod tuple.

pull-kubernetes-unit reconciler TestRunOneVolumeDetachFailNodeWithReadWriteOnce unknown, maybe recently? No response /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The test is still flaking but it is almost rare. Please see the link below for the test history. The root cause is that the detach operation is not triggered or completed within 100ms. 1) not triggered: 2) triggered but not completed: It seems no better way to fix it unless we increase sleep time in the test. i.e. (100ms -200ms) I'm not sure if it is valuable to fix it. cc what do you think?
any flake should be fixed even if it's rare.
/assign


Question: metadata: name: %s unknownMeta: unknown <del> finalizers: - test-finalizer </del> spec: template: apiversion: foo/v1

After running k8s conformance testing on kubernetes v1.27, we have leftover custom resources of the format that should not exist. For example) Basic test instructions: We use the latest official kubernetes kubekins image: We extract kubetest binary from kubekins latest-master docker image. We run serial conformance tests as follows. We also run parallel conformance tests as follows.
/sig api-machinery
The following script is used to cleanup the leftover CRDs:
/triage accepted /assign /cc
/area conformance
/sig architecture


Question: package network import ( <del> "bytes" </del> "context" "encoding/json" "errors"

Example failure: Stacktrace: The problem seem to be related to the fact that we're running the test on private cluster (nodes doesn't have external IP address).
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Given that there are more issues with the NetLB tests than just this particular one, I think we should revert my PR that re-enabled them. ! Wojtek just sent a PR to do that -
But let's leave this issue open - we want to fix and re-enable it again.
The revert merged, but we didn't managed to get it in before the yesterday's run started. Today's run should be green.
Yeah - but that doesn't mean we should close it later - we should fix and reenable this test. The issue should be opened until that test is reenabled.
That's clear, I didn't say anywhere that we should close it
Ok, the tests have been disabled and they don't fail the correctness tests anymore. When it comes to fixing this particular test. I took a cursory glance and I think it is fixable. We should change the code to not expect public IPs on the nodes. And then when poking over http we should do it not from the test but proxy via some container. There is already a code written for that, see /help wanted
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I would like to take this /assign
- thanks!

contains brief description on how to fix that
so are we going to access all the nodes via a test container?
Not sure if I fully understand your question, so if you meant something different let me know: to make the test work in large clusters (5k nodes), we can't access all - we may want to do tricks we do in other tests like access min(cluster size, some constant (e.g. 10 or 100) if you meant "via a test container" - yes, I think it's fine to just migrate the test to always use it
Reopening - since we need to reenable this test.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Does this issue need an owner? I got a little lost wading through all of the issues/PRs attached to this issue. If I wanted to take up the reins, how do I trigger the 5k node test on a PR or is it automatically triggered?
The test is disabled. I think the crucial part is done now - which is: I can't remember what else left to be done. Unfortunately you can't do that from oss. We only have post-submits exposes as these are too expensive. We can try running it internally to check if it now works...
We reenabled this test and it seem to work. So closing this one. Thanks for all the help on it!


Question: } // NVIDIADevicePlugin returns the official Google Device Plugin pod for NVIDIA GPU in GKE <del> func NVIDIADevicePlugin(ns string) *v1.Pod { </del> <ins> func NVIDIADevicePlugin() *v1.Pod { </ins> ds, err := DsFromManifest(GPUDevicePluginDSYAML) Expect(err).NotTo(HaveOccurred()) p := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "device-plugin-nvidia-gpu-" + string(uuid.NewUUID()), <del> Namespace: ns, </del> <ins> Namespace: metav1.NamespaceSystem, </ins> }, Spec: ds.Spec.Template.Spec,

https://k8s- The following tests are broken: NVIDIA GPU Device Plugin PriorityLocalStorageEvictionOrdering PriorityMemoryEvictionOrdering Looks like some new validation was to prevent two things: PodPriority directly system-critical priority classes outside of the kube-system namespace. To fix this, we need to create new priority classes for these tests. cc /kind failing-test /sig node /sig scheduling /priority important-soon
/kind bug
/assign
This doesn't appear to have fixed the tests. Tests now panic during the Aftereach in which the PriorityClass is deleted. /reopen
Panic:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: bandwidthBits int64 } <ins> func (i *IPerfCSVResult) bandwidthMB() int64 { return int64(math.Round(float64(i.bandwidthBits) / float64(megabyte) / 8)) } </ins> // Add adds a new result to the Results struct. <del> func (i *IPerfResults) Add(ipr *IPerfResult) { </del> <ins> func (i *IPerfResults) Add(ipr *IPerfCSVResult) { </ins> if i.BandwidthMap == nil { i.BandwidthMap = map[string]int64{} }

What happened: The iperf tests were written a while ago but i think havent been updated. They measure bandwidth over multiple pods, but they probably should also have a test which reports individual pod's bandwidth between one pod over a SEP. They should pretty print a table so its easy to see exactly what is going on with the bandwidth They should use iperf3 (need to check that) Maybe we should have permutations for udp and tcp? The current output is something like: What you expected to happen: That running the tests would result in a table such as those that you can get from iperf3, i.e. How to reproduce it: run the e2es on any cluster, Environment: Kubernetes version (use ): master, any version
/sig network
/assign
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I've taken a first pass at this , working on resolving a couple issues then we can circle back to determine how else it could be improved!
Cool c/p the new test output so we can compare when u get a chance
ah yess. much better then what we have now.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: <ins> Copyright ©2013 The gonum Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the gonum project nor the names of its authors and contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. </ins>  No newline at end of file

we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?


Question: // during the test so clean it up here rc, err := c.ReplicationControllers(ns).Get(RCName) if err == nil && rc.Spec.Replicas != 0 { <del> DeleteRC(c, ns, RCName) </del> <ins> By("Cleaning up the replication controller") err := DeleteRC(c, ns, RCName) expectNoError(err) } // Clean up the namespace if a non-default one was used if ns != api.NamespaceDefault { By("Cleaning up the namespace") err := c.Namespaces().Delete(ns) expectNoError(err) </ins> } }) // Tests with "Skipped" substring in their name will be skipped when running // e2e test suite without --ginkgo.focus & --ginkgo.skip flags. type Density struct { skip          bool podsPerMinion int

Currently the density tests are ok if all the pods come up eventually. We should verify there are no failure/error events while the pods are coming up. This could indicate issues dealing with the number of pods and should be investigated.
has been merged, and there's already an issue filed for followup actions, so I think we can close this.


Question: "k8s.io/apimachinery/pkg/watch" "k8s.io/apiserver/pkg/apis/example" examplev1 "k8s.io/apiserver/pkg/apis/example/v1" <ins> example2v1 "k8s.io/apiserver/pkg/apis/example2/v1" </ins> "k8s.io/apiserver/pkg/features" "k8s.io/apiserver/pkg/storage" "k8s.io/apiserver/pkg/storage/etcd3"

The test is flaking: /assign /sig api-machinery /priority important-soon
Ok - it's fairly easy to reproduce by just adding into the test I will validate the fix.
/triage accepted


Question: j.Service = &empty } }, <ins> func(j *core.LoadBalancerStatus, c fuzz.Continue) { ipMode := core.LoadBalancerIPModeVIP for i := range j.Ingress { if j.Ingress[i].IPMode == nil { j.Ingress[i].IPMode = &ipMode } } }, </ins> } }

ci-kubernetes-unit testing Failed; === RUN TestRoundTripTypes since 02-25 No response /sig api-machinery network
It started to flake after /cc
It is also flaking in our internal CI on
/triage accepted


Question: if evaluator == nil { listerFunc := generic.ListerFuncForResourceFunc(qm.informerFactory.ForResource) listResourceFunc := generic.ListResourceUsingListerFunc(listerFunc, resource) <del> evaluator = generic.NewObjectCountEvaluator(false, resource.GroupResource(), listResourceFunc, "") </del> <ins> evaluator = generic.NewObjectCountEvaluator(resource.GroupResource(), listResourceFunc, "") </ins> qm.registry.Add(evaluator) glog.Infof("QuotaMonitor created object count evaluator for %s", resource.GroupResource()) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig api-machinery What happened: The that we use to make a create validation function is given admission.Update as its operation field. In the create validation we then use the provided operation (admission.Update) , when we call the admission controllers, even though a new object is being created. A similar thing is happening with the mutating admission. This means that when an objects is created on update, the admission controllers will see this as an update request. Meaning there could be an objects in storage which never passed create admission validation and were never modified by create admission mutation. I think this is a bug. What you expected to happen: Admission controllers would handle a create-on-update exactly the same as a normal create
I think the solution to this would be to separate the static admission attributes into one for the create validation and one for the update validation. Also, the function which performs the mutating admission needs to determine whether or not it is an actual update or if the old object doesn't exist (probably by checking uid) and then call the mutating admission correctly based on which operation is being performed.
/assign I might be interested to give it a try.
How to reproduce this bug?
Instead of a reproduceable BUG, I think it's like a mismatch of conceptual design and actual implementation. If a resource allows create-on-update in its strategy e.g.: Then admission plugins would receive operations (CREATE/UPDATE) on UPDATE requests. Some problems might happen the admission plugin is written without considering the circumstances that admitting a UPDATE operation might result a object creation into the storage. Risky and confusing, WDYT?
With the current build in admission controllers, I'm not sure if there are any bugs. Maybe something with resource quota? The real problem is that there is no way to know which webhook authors will have a dependency on seeing a "create" operation for every object put into storage. If a webhook relies on creates to implement its custom logic (which should be allowed), then objects created with an update would currently be missed by the webhook.


Question: REGISTRY ?= staging-k8s.gcr.io IMAGE = $(REGISTRY)/pause <del> TAG ?= 3.7 </del> <ins> TAG ?= 3.8 </ins> REV = $(shell git describe --contains --always --match='v*') # Architectures supported: amd64, arm, arm64, ppc64le and s390x

The Windows pause images is built using buildkit on Linux hosts and this configuration does not allow for the Windows registry to me modified during the container build process. When Windows containers are built on Windows host machines part of the build process creates 'delta' registry files when get stored in the layer files. These 'delta' registry hive files are applied to the containers active registry when the containers are created. Chaning the registry files in the container image layers without having the 'delta' files present will result in the changes NOT being present in the container image. We have an increasing demand for the ability to modify registry in the pause image on Windows such as Disable some services like DiagTrack that negatively impact performance Change behavior of RPCSS so it doesn't hijack RPC port (135) for all containers in the pod Things are working as expected given the current build process. There are linux utilities like that can modify Windows registry files that I have used to test this out. After running the container produced from these files you can see the registry changes made by persisted in the container image but these changes are not present in the container's active registry. vs One possible solution is to create an intermediate image based on (the current base image for the Windows pause image) that performs all the desired registry operations that is built on a Windows machine and published someplace (MCR maybe?) and the use that as the container base image for the Windows pause image. <detailsN/A </details<detailsN/A </details<detailsN/A </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
cc
my first concern was. will a community member be able to build this image themselves (will the sources be available and up-to-date and the image built from it). We could also do a github hosted windows runner to build / publish this new image as well. let's please revisit this once we wrap 1.24.

/milestone v1.25
We created which has the source/docker files used to build . AKS has made a pause image based on image available to some customers out of necessity to unblock some scenarios. We'd like to upstream the changes - and do so with as little destabilization to Kubernetes project as possible.
Seems like the intermediate base image step would be reasonable. I don't think community members can build the existing base windows images either for the same reason of depending on windows?
The windows pause image is currently based on 'nanoserver' which is one of several container base images provided by Microsoft (and only Microsoft can produce container base images).
yep. i'd expect community downstream teams (for example) would like to start with this nanoserver, have a set of dockerfiles to build this new image ... then build their own pause images based on this new image
or in other words if MSFT wants to roll all the changes into nanoserver itself (since we all already know that only MSFT can produce this), then there is no debate after that
We can't roll these changes into nanoserver because they may break other scenarios. Because of that - having an intermediate layer built somewhere was the best solution we came up with without forking the build process for the pause image. Microsoft is willing to host build/host the intermediate layers/images from open-sourced docker files. If we want to build the intermediate layers somewhere else too (like github hosted Windows runners) that would be fine too.
/sig windows
If this intermediate image used only by k8s then it is a good idea to build the image via GH actions in the kubernetes org so that k8s maintainers have better control on this or if this image is going to be generic for anyone to use then MSFT can build one and host it somewhere.
Picking this back up since 1.24 release is behind everyone. - Do you think a separate github repo for the dockerfile and github actions staging/prod container repos would be appropriate here? If so I can get started on creating these. Also - Is it possible to push to the staging repos from a github action job instead of PROW jobs? We'd need to use github actions to build the Windows image on a Windows machine.

I supposed we can store the .tar files as release artifacts if needed and then use a PROW to job push them to the staging repo if needed but being able to push for the action would be simpler.
/cc as FYI
I see two possible paths forward Continue to host the source for the intermediate layer at , publish the image to , and then update the Windows pause build docker files in k/k to consume this image instead of the or Create a new kubernetes-sigs repo to host the source and build steps for the image, publish the image to , and then update the Windows pause build docker files in k/k to consume this image. I'm fine with either approach but am not sure who the right folks to discuss this with are.
This looks more promising because this image you are creating is not a specific change for the pause container, IMO if we can change that image name to something generic and use the same wherever required.
I think those both sound reasonable personally. I think it's technically feasible to push from GitHub actions to staging GCR with workload identity but nobody has done this yet in Kubernetes.


Question: "net.ipv4.ip_unprivileged_port_start", "net.ipv4.ip_local_reserved_ports", "net.ipv4.tcp_keepalive_time", <ins> "net.ipv4.tcp_fin_timeout", "net.ipv4.tcp_keepalive_intvl", "net.ipv4.tcp_keepalive_probes", </ins> ) )

Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted


Question: able to start with Secret and ConfigMap volumes mounted into the container. release: v1.13 file: test/e2e/storage/empty_dir_wrapper.go <ins> - testname: PersistentVolumes(Claims), lifecycle codename: '[sig-storage] PersistentVolumes CSI Conformance should run through the lifecycle of a PV and a PVC [Conformance]' description: Creating PV and PVC MUST succeed. Listing PVs with a labelSelector MUST succeed. Listing PVCs in a namespace MUST succeed. Patching a PV MUST succeed with its new label found. Patching a PVC MUST succeed with its new label found. Reading a PV and PVC MUST succeed with required UID retrieved. Deleting a PVC and PV MUST succeed and it MUST be confirmed. Replacement PV and PVC MUST be created. Updating a PV MUST succeed with its new label found. Updating a PVC MUST succeed with its new label found. Deleting the PVC and PV via deleteCollection MUST succeed and it MUST be confirmed. release: v1.29 file: test/e2e/storage/persistent_volumes.go </ins> - testname: Projected Volume, multiple projections codename: '[sig-storage] Projected combined should project all components that make up the projection API [Projection][NodeConformance] [Conformance]'

[X] test approval issue : [X] test pr : 26 May 2023 [x] two weeks soak end date : 9 June 2023 [x] test promotion pr : [x] remove endpoint from [x] End of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection. According to following APIsnoop query, there are still 3 CSIDriver endpoints that are not tested for Conformance. Using a number of existing e2e test practices a new has been created to address these 3 endpoints. The e2e logs for this test are listed below. This query shows the following csiDriver endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen For end of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection.
Reopened this issue. <detailsIn response to : /reopen For end of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # GLBC ensures that the `cloud.google.com/backendconfigs` and `networking.gke.io/servicenetworkendpointgroups` CRD exists in a desired state: # https://github.com/kubernetes/ingress-gce/blob/5c3fcb5845e74b92ea8bd52929b15fc5c9fa7970/cmd/glbc/main.go#L108 # https://github.com/kubernetes/ingress-gce/blob/5c3fcb5845e74b92ea8bd52929b15fc5c9fa7970/cmd/glbc/main.go#L133 <ins> # GLBC creates and updates `networking.gke.io/GCPIngressParams` # https://github.com/kubernetes/ingress-gce/blob/7f0928629c85e7a54c6af9e6e490ac89d057461a/cmd/glbc/main.go#L151-L162 </ins> # TODO(rramkumar1): https://github.com/kubernetes/ingress-gce/issues/744 - apiGroups: ["apiextensions.k8s.io"] resources: ["customresourcedefinitions"]

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /cc /sig network /priority important-soon /milestone v1.20
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
xref
/assign
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has been green for a while now. Should we close this one?
I would ideally like to see a statement that this was triaged, had a root cause analysis done, had a fix for the root cause applied, and that the fix was monitored and deemed successful. Hey can you take a look at let us know if you are happy to that this flake has been figured out and fixed? is correct, this looks green. but this does not
Flaked today: And triage also shows that the version of this test has been having a bad time for a long time on :
I think there was a CI change that caused this to go green. Will follow up.
I am looking at this test grid. It is relatively green. Am I looking at the right one?
Yes, that's the one where this one is. But there is also a one that Rob and I are referring to which is here: and has not passed for a while.
Can we remove this from SIG-Net in favor of GCE or GKE cloudprovider? If this is mostly to due with NEG, it likely only applies to the GKE specific test.
What do we do with this one?
this continues to fail hundreds of times a day - if the thing it is testing is important, should this be prioritized? if not, should the test be removed?

we have several of these, and despite they are tagged as sig-network most of them seems to be GCE only tests. I'm tagging you because you'll have better visibility. Maybe we should define clearly the ownership, is this sig-network, gce or cloud-provider? and then the owner decide if we want to run them or fix them, leaving tests failing in this limbo is not useful as Jordan correctly pointed out
+100 for not staying in that state - we should either prioritize or remove. Re ownership - I don't have strong opinion here. But for sure the name is misleading if the tests are GCE-specific (I would expect "conformance" stuff to not be gce-specific. What are the arguments against 'sig-network' owning it? It's just that you don't want to debug GCE-specific stuff? Or some more fundamental ones (like ability to iterate etc.)?
I didn't dig too much into the tests because I always assumed that was testing GCE only things, but it is possible that I'm wrong. I personally don't have problem on debugging GCE stuff, is the ability to iterate and the codebase that is being tested that worries me: is any person able to debug the test? are there parts of the test that need "internal GCE" access to debug it? is the test testing kubernetes network or gce cloud-provider, gce ingress, ....?
I think that everything should be accessible - to confirm. Technically, many networking (and other) tests are relying on cloud-provider (e.g. all tests that exercise LoadBalanced service). So that isn't unique (do we have complaints about these tests too)? So what make ingress tests different is a separate out-of-tree controller, I guess. [To be clear - I'm not trying to push sig-network to own that, i'm just trying to understand the arguments better.]
Let me share my thoughts, I'm personally trying to make all the dashboards in sig-network green, and this week I was able to sort out most of the problems, the last flake with TCP CLOSE_WAIT seems solved, and everything is acceptable except in GCE and Azure jobs.

The dual-stack azure jobs are pending of a new image version bump, but I know there is an Azure person that is taking responsibility and will be working on fixing it: For GCE jobs I use to ping or and you can see that the gce dashboard is in a good state However, there are GCE jobs that are in very bad state, failing continuously, I'm practically doing this by myself, that is a personal decision, spending some time every day to review the dashboards and addressing the "generic" failures, but my time is limited, so if this has to go under sig-network umbrella I just want to be sure that these are really needed and, if they are GCE only features, asking for some help from GCE folks.
I agree we're relying to much on heroics of individual people that isn't sustainable. I think those are valid questions, but I would first like to hear from network folks with Google too.
is taking a look
This might just be a issue with 10m not being quite enough for 100% of the time to pass.
Strike my comment -- looks like there is test related issue.
The tests are failing because glbc needs new rbac permissions for IngressClass and GCPIngressParams. adds those which should cause the the failures to stop.
/assign Thanks guys, we are close to be completely green in sig-network :smile:
to fix the
/reopen It seems only 2 tests were fixed, there are still 7 failing tests
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
maybe this error failure gives you a hint
seem to show another rbac issue:
patch permission to services/status has to be granted separately
adds the permission
Is this resolved yet? I just saw another failure today -
The most recent failure does not look to be because of rbac permissions. But I realize this issue is tracking multiple errors and this error looks more similar to the original problem described. The rbac permissions in has not been merged in yet.

/milestone v1.21 (marking in the current milestone until we get confirmation whether it is relevant to the release or not)
Slack Discussion Thread:
I have been looking at the test failure: This failure actually is a different test than the one this issue was opened with. Looking at the logs, there does not seem to be a clear indication of the root cause. The error is because the loadbalancer comes up within the timeout but the path has not be configured leading to the 404. Looking at the logs, the error seems to be outside of the Kubernetes release and therefore this flake should not be considered as release blocking.
/milestone clear (since this has been deemed to be non release-blocking)
Current logs are not enough to determine the issue. For now, opened to update the glbc image which should give us more instrumentation. The timeout is currently 10 min. One theory is that this sometimes is not enough for the loadbalancer to be ready. The LB comes up, but the path is not available yet. However since the the timeout is currently shared amongst many different tests, I propose we wait until we can determine that is the root cause before increasing the timeout.
this is much better now, only one failures in the last 10 days and the failures is as you said:
Can we update the timeout to 2x (20 min) -- I think this will drop failures significantly.
The timeout is set in a helper method: and uses: Are we okay changing it for all the tests that use that, or do we only want to target this one?
Let's target it for this test -- as I think this is the only one that will require such a long timeout.
:upsidedownface: if you want to be on the safe side :) :)
This still needs more work, while working past week on the ingress v1beta1 problems I found that sometimes the client that polls the LB hangs, we should at least have a better logging to understand which part is broken on the test. Is the client? the LB? something between both?
-- any code reference? Improving the client logging sounds like an easy PR
It failed twice the last week, but now is failing in another point can you PTAL, what is that certificate error?

/milestone v1.22
it only failed once, last week, however, the job is running every 30 min that seems excessive. I'll update the job to run with a larger interval and we'll keep monitoring it
Decreasing period sounds good to me.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: opts: nil, } } <del> testManager.allDevices[res.resourceName] = makeDevice(res.devs) </del> <ins> testManager.allDevices[res.resourceName] = makeDevice(res.devs, res.topology) </ins> } return testManager, nil

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: Can't find this in Triage Example spyglass links: /sig scheduling /cc /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't quite think this test should be owned by SIG-Scheduling. Most of the failures happen in the Nvidia GPU setup phase. BTW: although irrelevant, last time we see a failure of this test is .
Searching keyword "UnexpectedAdmissionError" gets the following occurrences:
/sig node cc in case you're aware of any recent changes that may have caused this failure.
From the I can see a bunch of: Checking the kubelet log, I can see the gpu capacity has been changing And before the capacity changed to 0, we can see the kubelet was attempting to allocate 1 GPU to the test pod, but some NPE was thrown, and kubelet stopped/restarted.
The nil pointer is from here: , which is a new change introduced by
The current device plugin (the one defined ) is using an old version of device plugin api (see ), which does not have field under . So calling will end up with NPE.
Thanks , the analysis makes great sense to me. Could you please work on a fix to address this? (Probably updating )
Hi , I am still looking at the code, because the code does have the logic to handle the case where device plugin does not return affinity: Also, even if the Topology is not defined, I am not sure will throw NPE () (please correct me if I am wrong here). I will need to look more closely at the code to find out where the NPE is from. Updating the might take long time because we will need some testing to make sure it doesn't break anything.

Currently I am thinking of identifying the source of NPE, and make sure can handle that case.
/cc
I submitted a fix. My fault, it was a real case when pointer to Topology could be a nil. But still fails: I don't observe SIGSEGV now in , but 3 problems there: 1113 09:26:19. 1336 ] failed to run Kubelet: cannot create certificate signing request: is forbidden: User "kubelet" cannot create resource "certificatesigningrequests" in API group "" at the cluster scope of nvidia-driver-installer stops and restarts after a while. 's () NoopManager is using
can you show full error msg of "NVIDIA GPU metrics will not be available: %s"?
I1113 15:43:26. 1338 ] Found device with vendorID "0x10de" W1113 15:43:26. 1338 ] NVIDIA GPU metrics will not be available: Could not initialize NVML: could not load NVML library I1113 15:43:26. 1338 ] Machine: {Timestamp:2020-11-13 15:43:26. +0000 UTC m=+5. NumCores:2 NumPhysicalCores:1 NumSockets:1 CpuFrequency: MemoryCapacity: MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:2048 NumPages:0}] MachineID: SystemUUID:-9824-3a64-f41f- BootID:-f891-4c0b-a34d- Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:39 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/sda8 DeviceMajor:8 DeviceMinor:8 Capacity: Type:vfs Inodes:4096 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:19 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true}

{Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:22 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/etc/machine-id DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/mnt/disks DeviceMajor:0 DeviceMinor:38 Capacity: Type:vfs Inodes: HasInodes:true} {Device:overlayfs0-43 DeviceMajor:0 DeviceMinor:43 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/var/lib/cloud DeviceMajor:0 DeviceMinor:45 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/root DeviceMajor:253 DeviceMinor:0 Capacity: Type:vfs Inodes: HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size: Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size: Scheduler:mq-deadline} 9:0:{Name:md0 Major:9 Minor:0 Size:0 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:42:01:0a:28:00:03 Speed:-1 Mtu:1460}] Topology:[{Id:0 Memory: HugePages:[{PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 1] Caches:[{Size: Type:Data Level:1} {Size: Type:Instruction Level:1} {Size: Type:Unified Level:2}] SocketID:0}] Caches:[{Size: Type:Unified Level:3}]}] CloudProvider:GCE InstanceType:n1-standard-2 InstanceID:} I1113 15:43:26. 1338 ] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. I1113 15:43:26. 1338 ] Version: {KernelVersion:5.4.49+ ContainerOsVersion:Container-Optimized OS from Google DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:} I guess kubelet can't load in cAdvisor the NVML library.
I have looked into this issue actually.

This is caused by a recent change on cadvisor: We used to return a real nvidiaManager but now only a Noop one if the NVML lib is not there. And currently the NVML lib is loaded by daemonset, which will not there when the kubelet starts. This PR should fix it: But btw, this bug will not fail the test because it will only cause missing GPU usage metrics. The plan is to cherry-pick the fix above to cadvisor 0.37, and then update the vendor for kubernetes


Question: return volumetypes.GeneratedOperations{}, volumeToDetach.GenerateErrorDetailed("DetachVolume.SplitUniqueName failed", err) } attachableVolumePlugin, err = og.volumePluginMgr.FindAttachablePluginByName(pluginName) <del> if err != nil { return volumetypes.GeneratedOperations{}, volumeToDetach.GenerateErrorDetailed("DetachVolume.FindAttachablePluginBySpec failed", err) </del> <ins> if err != nil || attachableVolumePlugin == nil { return volumetypes.GeneratedOperations{}, volumeToDetach.GenerateErrorDetailed("DetachVolume.FindAttachablePluginByName failed", err) </ins> } }

What happened: During investigation of nil pointer dereference panic, i've found that has an issue: don't check 1st return value of to : but could return : So, if finds plugin by name, but this plugin isn't a , it will cause a nil pointer dereference panic in What you expected to happen: should check How to reproduce it (as minimally and precisely as possible): Unfortunately, i can't reproduce it Anything else we need to know?: advised to change signature to <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig storage
how often have you run in to this? can you help put together a fix?
We could fix this but IMO this should never be reproducible because this panic will only happen if function finds a plugin but the plugin is not attachable. This could never happen because we only initialize attachable volume plugins in attach/detach controller and hence will ALWAYS return error because it will not be able to find a non-attachable plugin with that name.
Yes that makes sense. We need more info from Also do you have any modification or are you running a standard k8s?
it could be argued that - we should still fix this for sake for clarity. I will do that post 1.13. /assign
I've sent a PR to fix this issue: We it in about 30 minutes during rolling update, and, unfortunately, we're unable to reproduce it. It was k8s with modified apiserver & kubelet, but there was no modifications in kube-controller-manager. Panic trace:


Question: if err != nil { t.Errorf("%v: error %v (%#v)n", test, err, err) } <del> if test != lq.String() { </del> <ins> if strings.Replace(test, " ", "", -1) != lq.String() { </ins> t.Errorf("%v restring gave: %vn", test, lq.String()) } }

When polling a k8s master with the response is an HTTP 500. But even if this selector is invalid, it should return a 4xx code instead.
Do you get 500 without the label selector? cc
No, and not with a value like "environment%3Dfoobar". But if I remove "foobar" or replace it with an url-encoded space, i.e. "environment%3D%20" or "environment%3D+" I get a 500. I suspect that the value (after the colon) is stripped of whitespace, and if the result is the empty string, this error happens.
Bumping priority to P1 since this is definitely a bug and we should fix it.
you had five days, I'm kicking you off of this & giving to Chao. :)


Question: return cmdutil.AddSourceToErr("creating", info.Source, err) } <del> if cmdutil.ShouldRecord(cmd, info) { if err := cmdutil.RecordChangeCause(info.Object, f.Command(cmd, false)); err != nil { return cmdutil.AddSourceToErr("creating", info.Source, err) } } </del> if !dryRun { // Then create the resource and skip the three-way merge if err := createAndRefresh(info); err != nil {

What happened: What you expected to happen: DS history revision 2 should have change cause from the second command "kubectl apply -f ds-2.yaml --record", not the first /assign


Question: COPY peer-finder / # See README.md <del> RUN wget -q -O /zookeeper-3.5.0-alpha.tar.gz http://apache.mirrors.pair.com/zookeeper/zookeeper-3.5.0-alpha/zookeeper-3.5.0-alpha.tar.gz &&  </del> <ins> RUN wget -q -O /zookeeper-3.5.0-alpha.tar.gz http://archive.apache.org/dist/zookeeper/zookeeper-3.5.0-alpha/zookeeper-3.5.0-alpha.tar.gz &&  </ins> tar -xzf /zookeeper-3.5.0-alpha.tar.gz -C /tmp/ && mv /tmp/zookeeper-3.5.0-alpha /zookeeper && rm /zookeeper-3.5.0-alpha.tar.gz ADD install.sh /

/kind bug What happened: The test image pets/zookeeper-installer references an old alpha version of zookeeper: This can't be built anymore, as it no longer exists on the server: Sure enough, now lists beta versions for 3.5.x, as well as stable 3.4.x versions, but no 3.5.0 alphas. We probably should update this to a newer version, or alternately delete the test and image if it's no longer providing signal. How to reproduce it (as minimally and precisely as possible): /sig apps


Question: attach, err := c.k8s.StorageV1beta1().VolumeAttachments().Get(attachID, meta.GetOptions{}) if err != nil { glog.Error(log("attacher.WaitForAttach failed for volume [%s] (will continue to try): %v", volumeHandle, err)) <del> return "", err </del> <ins> return "", fmt.Errorf("volume %v has GET error for volume attachment %v: %v", volumeHandle, attachID, err) </ins> } // if being deleted, fail fast if attach.GetDeletionTimestamp() != nil {

brought this up during the CSI Standup. It seems to be true, in the test config we do not set which is required to bring up the pod for consuming the volume. The code where is checked: The test config used where is not set to true: /kind bug /sig storage /assign /cc


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/client-go/pkg/api/v1" <ins> appsv1beta1 "k8s.io/client-go/pkg/apis/apps/v1beta1" </ins> ) // describes the attributes of a scale subresource

Here are the hash label keys of all controllers: Deployment: DaemonSet: (new in 1.7) StatefulSet: (new in 1.7) Had a brief discussion with and about this, it should be consistent. We can use for DaemonSet and StatefulSet, and keep Deployment as-is. Deployment hash label key should be different from other two because Deployment pods hash labels is used by selector (unlike the other two controllers). We probably should have used hash annotations instead of hash labels for DaemonSet and StatefulSet, since we don't use hash labels to select their history or pods. Hash labels are chosen for consistency. This is fine since we don't modify hash label values.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Actually, controller revisions created by statefulset, daemonset has different label: // controller revision created by statefulset label: "" // controller revision created by daemonset label: controller-revision-hash: "" should be consistent?


Question: return fmt.Errorf("rbd: failed to mkdir %s, error", globalPDPath) } <ins> // fence off other mappers if err := util.fencing(rbd); err != nil { return fmt.Errorf("rbd: image %s is locked by other nodes", rbd.Image) } // rbd lock remove needs ceph and image config // but kubelet doesn't get them from apiserver during teardown // so persit rbd config so upon disk detach, rbd lock can be removed // since rbd json is persisted in the same local directory that is used as rbd mountpoint later, // the json file remains invisible during rbd mount and thus won't be removed accidentally. util.persistRBD(rbd, globalPDPath) </ins> if err = rbd.mounter.Mount(devicePath, globalPDPath, rbd.fsType, nil); err != nil { err = fmt.Errorf("rbd: failed to mount rbd volume %s [%s] to %s, error %v", devicePath, rbd.fsType, globalPDPath, err) }

etcd cluster backing apiserver died spontaneously, after restart apiserver pulled up the pods on different machines. Node storage was backed by ext4 over rbd, which became mounted on several nodes at once. That could have possibly led to data corruption. I guess that the proper way to mount rbd volume is to take a lock on it first, which will at least fail if it's locked somewhere else, requiring manual intervention, but not destroying data. The lock should be optional, in case a proper cluster fs is mounted over the rbd. In this case there was clearly no need for accessing the data from several pods so ext4 is deemed a proper fs.
, Vladimir Pouzanov < :
want to get some detail. How is your pod-to-rbd mapping like? Since rbd image is a block device, it can be used by one pod at a time. I am not sure if locking is a proper way to go - if the pod holds a lock but dies in the middle of mounting, the rbd image (or any other volume type) is then excluded from access. In such case, a high level coordination at apiserver is a better place to resolve this problem.
Here's my rc: I guess it's better to have a lock to be cleaned up manually rather than to have non-cluster fs to be mounted in several locations at once?
thanks, this pod looks good to me. what would cause multiple nodes run the same pod?
I don't know the exact reason of why kubelet didn't unmount the old fs, but after apiserver and scheduler were restarted pod got allocated on different node.
It is a scary thing to see etcd down, it is meant to be reliable...
Unfortunately I don't have long-term logging in test cluster so I have no idea why it actually died. but using kubedns as primary node resolver was clearly a bad idea, once it died, nodes stuck with no resolver at all.
your thought on HA?
Wrong Tim St. Clair?
- you HAVE to be ready to handle network partitions - the old node might still be alive and even using the volume, but the apiserver can't see the node, so it thinks it deletes the pod and starts it elsewhere. If RBD corrupts itslef in this case, it is not useful and we should strip it from the codebase.

, Huamin Chen wrote:
that makes sense. rbd doesn't corrupt itself. This is inconsistency happens when you have two pods mounting the same rbd image. How does GCE PD handle/prevent this case?
It doesn't allow more than one read-write attachment. Multiple read-only attachments is OK. Better to fail than to allow corruption. , Huamin Chen wrote:
That's neat. I have a lock fencing for rbd, testing now.


Question: - --namespace=kube-system - --configmap=kube-dns-autoscaler # Should keep target in sync with cluster/addons/dns/kube-dns.yaml.base <del> - --target=Deployment/kube-dns </del> <ins> - --target={{.Target}} </ins> # When cluster is using large nodes(with more cores), "coresPerReplica" should dominate. # If using small nodes, "nodesPerReplica" should dominate. - --default-params={"linear":{"coresPerReplica":256,"nodesPerReplica":16,"preventSinglePointFailure":true}}

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).


Question: // SetGroupVersionKind sets or clears the intended serialized kind of an object. Passing kind nil // should clear the current setting. SetGroupVersionKind(kind GroupVersionKind) <del> // GroupVersionKind returns the stored group, version, and kind of an object, or nil if the object does // not expose or provide these fields. </del> <ins> // GroupVersionKind returns the stored group, version, and kind of an object, or an empty struct // if the object does not expose or provide these fields. </ins> GroupVersionKind() GroupVersionKind }

The statement "returns [...], or nil" cannot be true since the return type is a struct: Suggest it could say ", or an empty struct if ..." /sig api-machinery
I can file a PR to fix the incorrect comment if you don't mind :)
Sure, I didn't get round to it yet.
/assign


Question: type RollingUpdateDeployment struct { // The maximum number of pods that can be unavailable during the update. // Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). <del> // Absolute number is calculated from percentage by rounding up. </del> <ins> // Absolute number is calculated from percentage by rounding down. </ins> // This can not be 0 if MaxSurge is 0. // By default, a fixed value of 1 is used. // Example: when this is set to 30%, the old RC can be scaled down to 70% of desired pods

API documentation says is rounded up, but it is actually rounded down: The code shows it is rounded down, not up: and
Yes, the documentation is obsolete. We changed maxUnavailable to round down so that we wouldn't accidentally infer 100% unavailability for non-100 percentages. For example, maxSurge=25% and maxUanavailable=25% with a Deployment of 1 replica meant that we could scale down 1 before thus leading in 100% maxUnavailable (implicit).


Question: --tags "${NODE_TAG}"  ${network}  $(get-scope-flags)  <del> --metadata-from-file "$(get-linux-node-instance-metadata-from-file)" </del> <ins> --metadata-from-file "$(get-node-instance-metadata-from-file)" </ins> } # Assumes:

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The gce cluster creation script fails, when HEAPSTERMACHINETYPE env variable is provided. The script calls What you expected to happen: Script shouldn't fail. How to reproduce it (as minimally and precisely as possible): Run hack script to setup the cluster. Anything else we need to know?: Tis bug was introduced in This problem is due to call to not existing function - get-linux-node-instance-metadata-from-file": Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Linux Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Will be fixed by


Question: // mountPropagation determines how mounts are propagated from the host // to container and the other way around. // When not set, MountPropagationHostToContainer is used. <del> // This field is alpha in 1.8 and can be reworked or removed in a future // release. </del> <ins> // This field is beta in 1.10. </ins> // +optional optional string mountPropagation = 5; }


Question: "k8s.io/utils/clock" flowcontrol "k8s.io/api/flowcontrol/v1beta3" <ins> flowcontrolapplyconfiguration "k8s.io/client-go/applyconfigurations/flowcontrol/v1beta3" </ins> flowcontrolclient "k8s.io/client-go/kubernetes/typed/flowcontrol/v1beta3" flowcontrollister "k8s.io/client-go/listers/flowcontrol/v1beta3" )

Integration test timed out due to the following error The error appeared 748 times in the log. Looks like lots of churning happens during the apf configuration bootstrapping phase. Is there a race condition between when the type is being registered and the API calls? Stack trace: Integration job link: PR: I searched for this error in CI and this PR, looks like we only see this error in this PR so far. So I am hoping it has not introduced any flake. (maybe we need to let more time to pass to start seeing flakes) We should not see this error I saw it only in the PR mentioned.
/sig api-machinery cc
I see
The fact that it happens to a non-mandatory object implies that the object was received from the server, so the type must be in the server. The APF config-consuming controller does this:
Look for the output from this line:
Another PR with this failure:
the panic means the request scope FieldManager is nil that is initialized here: OpenAPIModels is set here: called from here: and openAPIModels is constructed here: it looks like it is possible for openapiconfig to be unset, which causes the openapi models to be nil, which causes apply to panic. I don't know if it is only the integration tests that are subject to that configuration. cc /priority important-soon (since this is now being exercised in bring-up of integration test servers)
if APF is creating/bootstrapping the objects without using apply, and then updating status using apply, that would explain how the object exists but apply fails
I suspect that APF is not successfully updating status at all in the integration tests with this server configuration
do you want me to revert , while we work on a solution?
I don't know how bad the test impact is. Work with / to determine if we should revert or fix forward
: I do not follow why you talked about server-side stuff then suggested the problem is in the client (a controller in the kube-apiserver) not supplying a fieldManager. The client does supply the fieldManager, for both creates and updates. See the following lines:
Different field manager. is set server-side in the endpoint handler based on the server configuration.
So if the problem is server-side, why did you suggest the problem is in the client?

Regarding the client, I looked in and found this line: actually several of the same form, all with the same fieldManager.
I don't think I did suggest that :) all of the setup and calls in are server-side
: please help me understand . That is about the client side. Are you saying that a client that creates an object using and later updates the object using --- with a non-empty fieldManager in both --- will run into this server-side scenario? If so, is that because this client behavior is wrong or because there is a bug server side?
That was in response to , trying to figure out how the object was created but could not be updated. Yes, when the server is not configured to enable openapi and therefore does not have apply enabled. Because the server is misconfigured (which, from what I can tell, is limited to some of the server instances started for integration tests)
Looking at and it is not obvious to me whether or how openapi could be disabled. Would that be by turning off the feature flag? FYI, the inclusion of "V3" in the flag makes me unsure what the server does about other versions. : I think you are saying that it is possible to disable openapi and that this in turn disables server-side apply. That seems kind of surprising to me. Does this mean that it is simply incorrect to run a client that does server-side apply with a server that has openapi disabled? Is this intended? Is it documented anywhere?
It's not intended as an externalized configuration option. integration tests have the ability to fiddle with many of the config struct options in the server that a kube-apiserver binary invocation could not. They intentionally use this to do things like inject specialized authorizers / authenticators for testing purposes, but also can unintentionally result in a test server that doesn't match a real API server invocation. I think this is an integration test server configuration problem, which should be fixed.
is correct that we have a misconfiguration in the integration tests and SSA is only enabled on a subset of integration tests. I'd like to understand the test failure conditions a bit better though. The can be consistently reproduced on a non SSA targeted test. (Eg: selected from one of the test failures). However, I'm seeing the panic message while the test is still green and passes locally.

it's a bit bizarre what contributes to the flakiness of the test because the FieldManager panic can be reproduced 100% of the time.
The panic occurs when APF is updating status (which it just switched to do using SSA). The APF status update failure does not block the API requests the test cares about, so the test passes. I could see extra CPU usage contributing to flakiness.
Thanks for the explanation. Created to fix the integration test configurations.
/triage accepted
The integration test job started timing out dramatically last night (50% of runs) once merged. Since that was the only change made to integration tests recently, it seems related. I think we should revert (and therefore ) until the test impact is understood
revert of -
did this get fully reverted? we're still seeing panics in integration tests over this:
bumping priority and milestoning until we know we're not currently triggering panics
looks like it got reintroduced in , revert open in
For integration tests, maybe we can cheat and cache the OpenAPI data, so that it's not recomputed for each test?
:-/ maybe... but I really don't want to have non-real init paths in integration if at all possible
I wish we could go a little further in lazily initializing openapi (especially v2), so let's see what we can do here (after code-freeze probably)
if SSA is actively using the openapi, it being lazy won't help, right?
does the revert mean that we can remove milestone v1.24 from this issue?
Yes, cleared
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Taking a fresh look at this PR and wondering if it has been mostly fixed already? From comments my understanding is that this panic occurred due to two conditions: running as part of the integration test suite is asked to service an SSA request while are nil. The apiservice has logic to panic in this case. From the code path commented is nil when the server is created with is nil. APF worker (running alongside an integration test) issues an SSA request to the test apiserver. Since the apiserver is misconfigured it panics. This panic could happen sporadically because sometimes the tests finish before the APF worker is able to issue the request. As a brief check to see if this might be fixed, I ran the test mentioned was failing from SSA. I logged out the 's created and found that they all had managed fields. I saw this as an indication SSA was now functioning properly. Indeed, to start the test server the function calls . This function has a code path which unconditionally leads to the apiserver's being set. calls before running the returned apiserver: calls before using the returned config to create the returned apiserver: uses which sets : Thus, any test using should no longer be affected by this panic. The question then becomes: do all integration tests start their servers in this way? It's hard to say. I found a PR (, ) which refactors many occurrences of the old stanza: Into something using the new method: ` But it appears is still being used in a few integration tests: Should these occurrences be refactored to use too?
There was a ton of changes I did around that. But the last PR (already removing the old is already out for review): - FYI Thinking about it now, I guess I agree that the problem might have been 10 different way kube-apiserver was set up through different tests.

I just don't know why it was causing flakes, not failing consistently...
There were definitely test setups that were not enabling openapi in the recent past. Two possibilities for flakes: Enabling openapi in all integration test setups was expensive and caused spurious timeouts The excessive numbers of background panics handled and logged in integration tests when the API server tried to handle SSA requests bogged down the CI machine (I think something like 40,000 panics with full stack traces each were getting logged)
i don't think that's the case. With the recent changes that I was doing as part of I think OpenAPI is already enabled for all tests having kube-apiserver (almost all) and that seems to work. But you remainded me that back then we also had an issue that we were running many more tests simulanuously, which could have been contributing to this problem significantly. The second reason is definitely possible contributor too. I guess my concern was that if some requests were panicing due to lack of OpenAPI, this means that these aren't really needed. But that actually might make sense - the fact that we weren't able to upgrade status of PriorityLevels/FlowSchemas shouldn't prevent non P&F-tests from succeeding, and that worked in P&F tests...
I think the panics were happening in async controllers, and no functionality blocked by the controllers intersected with the surface being tested, it just (maybe) caused log/perf issues
I believe has resolved the panic issue, I was able to verify with which changes the priority & fairness controller to use server side apply. Before merged, I analyzed the integration log: Integration test log file from an unrelated pr does not contain any trace of the panic as expected: On the other hand, the log file from integration job for (before merged) had plenty of these panics even though the integration job turned green: I wanted to check which tests produced these panics, so I did a breakdown by package Then I narrowed it down to the individual test(s): Before merged, the above tests used , and either is or is which results in to be nil. But a quick code search revealed that many more integration tests used , so why were the other tests not affected by the panic?

I inspected the log of a test that was not affected by the panic, and grepped for , and found the following: In the above case, the apf controller had not advanced far enough, the controller was active for about , may be these tests finish quickly and signals the server to shutdown before the apf controller can send an APPLY command. On the other hand, log from a test that was affected by the panic shows that the apf controller was active for more than , and has lot more activities: I am assuming that integration tests have at least log level 4. Also, the size of the log file differed significantly, without panics around , and with panics over and changed all these tests to use and now integration log for shows no panic.
Great! (This also proves why eliminating gazilion different things of initializing stuff in favor of one makes sense). - FYI ^^ - let's merge the PR then - I just one question there, so let's wait for Jordan to respond it.
Can we just force the openapi/fieldmanager to be set at start-time now that SSA is GA, it should basically always be present, rather than fail when SSA is unavailable (that's probably wrong by the way). If that triggers any test flakiness, then we would solve these. We can wait for to be solved since it will almost certainly have a huge impact on this.
Has this been done Jeff?


Question: NoDiskConflictPred = "NoDiskConflict" // PodToleratesNodeTaintsPred defines the name of predicate PodToleratesNodeTaints. PodToleratesNodeTaintsPred = "PodToleratesNodeTaints" <ins> // CheckNodeUnschedulablePred defines the name of predicate CheckNodeUnschedulablePredicate. CheckNodeUnschedulablePred = "CheckNodeUnschedulable" </ins> // PodToleratesNodeNoExecuteTaintsPred defines the name of predicate PodToleratesNodeNoExecuteTaints. PodToleratesNodeNoExecuteTaintsPred = "PodToleratesNodeNoExecuteTaints" // CheckNodeLabelPresencePred defines the name of predicate CheckNodeLabelPresence.

DaemonSet should not respect node's field. It's because cluster operators often want system components to be running and ready before workloads run on a given node. OTOH, DaemonSet does respect NoSchedule taints. This behavior is broken by , because DaemonSet share some predicates code with default scheduler.
/assign
Added this to 1.10 milestone, because it's a regression introduced in 1.10.
can you (or a SIG lead) add status/approved-for-milestone please? I'll add the priority. /priority important-soon
how is this use case satisfied by the daemonset controller creating pods and depending on the scheduler to assign them to nodes? having the scheduler ignore the unschedulable field seems nonsensical
I'd like get this as limitation for alpha feature; when upgrade to beta, I'd like to use taint to replace un-scheduable, so DaemonSet pod can tolerate it on-demand.
Escalating to critical-urgent, since we are now in Code Freeze. Please update daily with status, thanks! /priority critical-urgent /remove-priority important-soon
In short term, scheduler can check of pods and ignore bit of nodes for DS pods. Long term, when TaintNodesByCondition is stable, we should replace bit by taint and have DS controller add toleration for the taint when creating pods.
That doesn't seem proper. Making pods behave differently depending on the type of their creating object is very counterintuitive. That seems more reasonable. I'm less certain about auto- tolerations
- indeed there's some work currently to move from to a taint based behaviour, tracked by


Question: }, expectedElements: []string{"VolumeMode", "Block"}, }, <ins> { plugin: "local", pv: &api.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{Name: "bar"}, Spec: api.PersistentVolumeSpec{ PersistentVolumeSource: api.PersistentVolumeSource{ Local: &api.LocalVolumeSource{}, }, }, }, expectedElements:   []string{"Node Affinity:   <none>"}, unexpectedElements: []string{"Required Terms", "Term "}, }, { plugin: "local", pv: &api.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{Name: "bar"}, Spec: api.PersistentVolumeSpec{ PersistentVolumeSource: api.PersistentVolumeSource{ Local: &api.LocalVolumeSource{}, }, NodeAffinity: &api.VolumeNodeAffinity{}, }, }, expectedElements:   []string{"Node Affinity:   <none>"}, unexpectedElements: []string{"Required Terms", "Term "}, }, { plugin: "local", pv: &api.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{Name: "bar"}, Spec: api.PersistentVolumeSpec{ PersistentVolumeSource: api.PersistentVolumeSource{ Local: &api.LocalVolumeSource{}, }, NodeAffinity: &api.VolumeNodeAffinity{ Required: &api.NodeSelector{}, }, }, }, expectedElements:   []string{"Node Affinity", "Required Terms:  <none>"}, unexpectedElements: []string{"Term "}, }, { plugin: "local", pv: &api.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{Name: "bar"}, Spec: api.PersistentVolumeSpec{ PersistentVolumeSource: api.PersistentVolumeSource{ Local: &api.LocalVolumeSource{}, }, NodeAffinity: &api.VolumeNodeAffinity{ Required: &api.NodeSelector{ NodeSelectorTerms: []api.NodeSelectorTerm{ { MatchExpressions: []api.NodeSelectorRequirement{}, }, { MatchExpressions: []api.NodeSelectorRequirement{}, }, }, }, }, }, }, expectedElements: []string{"Node Affinity", "Required Terms", "Term 0", "Term 1"}, }, { plugin: "local", pv: &api.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{Name: "bar"}, Spec: api.PersistentVolumeSpec{ PersistentVolumeSource: api.PersistentVolumeSource{ Local: &api.LocalVolumeSource{}, }, NodeAffinity: &api.VolumeNodeAffinity{ Required: &api.NodeSelector{ NodeSelectorTerms: []api.NodeSelectorTerm{ { MatchExpressions: []api.NodeSelectorRequirement{ { Key:      "foo", Operator: "In", Values:   []string{"val1", "val2"}, }, { Key:      "foo", Operator: "Exists", }, }, }, }, }, }, }, }, expectedElements: []string{"Node Affinity", "Required Terms", "Term 0", "foo in [val1, val2]", "foo exists"}, }, </ins> } for _, test := range testCases {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: Now that it's a field, it needs a separate entry in 'kubectl describe'


Question: return false } } <del> completionMode := string(batch.NonIndexedCompletion) if isIndexedJob(job) { completionMode = string(*job.Spec.CompletionMode) } </del> job.Status.Conditions, _ = ensureJobConditionStatus(job.Status.Conditions, finishedCond.Type, finishedCond.Status, finishedCond.Reason, finishedCond.Message) if finishedCond.Type == batch.JobComplete { <ins> job.Status.CompletionTime = &finishedCond.LastTransitionTime } return true } // recordJobFinished records events and the job_finished_total metric for a finished job. func (jm *Controller) recordJobFinished(job *batch.Job, finishedCond *batch.JobCondition) bool { completionMode := getCompletionMode(job) if finishedCond.Type == batch.JobComplete { </ins> if job.Spec.Completions != nil && job.Status.Succeeded > *job.Spec.Completions { jm.recorder.Event(job, v1.EventTypeWarning, "TooManySucceededPods", "Too many succeeded pods running after completion count reached") } <del> job.Status.CompletionTime = &finishedCond.LastTransitionTime </del> jm.recorder.Event(job, v1.EventTypeNormal, "Completed", "Job completed") metrics.JobFinishedNum.WithLabelValues(completionMode, "succeeded").Inc() } else {

Occasionally job completions are counted twice (or potentially multiple times?) in the metric. This happens for about 20% of jobs, but may depend on the Job specificity, which results in an overestimation of the number of finished jobs. The metric reflects the actual number of completed jobs. a kind cluster with prometheus monitoring stack (for example follow this: ). the number of completed jobs by inspecting the metric and delete a job multiple times in a loop Example job: Example repro script: . Check the number of completed jobs by inspecting the metric and the job Yes, in the kube-controller-logs we find the following log lines which are related and probably indicate repeated updated which results in double counting. <details</details<detailsReproducible with kind </details<details</details<details</details<details</details<details</details
/assign
FYI
/sig apps
/triage accepted