Question: ginkgo.Context("with immediate allocation", func() { claimTests(resourcev1alpha2.AllocationModeImmediate) }) <ins> ginkgo.It("must deallocate after use when using delayed allocation", func(ctx context.Context) { parameters := b.parameters() pod := b.podExternal() claim := b.externalClaim(resourcev1alpha2.AllocationModeWaitForFirstConsumer) b.create(ctx, parameters, claim, pod) gomega.Eventually(ctx, func(ctx context.Context) (*resourcev1alpha2.ResourceClaim, error) { return b.f.ClientSet.ResourceV1alpha2().ResourceClaims(b.f.Namespace.Name).Get(ctx, claim.Name, metav1.GetOptions{}) }).WithTimeout(f.Timeouts.PodDelete).ShouldNot(gomega.HaveField("Status.Allocation", (*resourcev1alpha2.AllocationResult)(nil))) b.testPod(ctx, f.ClientSet, pod) ginkgo.By(fmt.Sprintf("deleting pod %s", klog.KObj(pod))) framework.ExpectNoError(b.f.ClientSet.CoreV1().Pods(b.f.Namespace.Name).Delete(ctx, pod.Name, metav1.DeleteOptions{})) ginkgo.By("waiting for claim to get deallocated") gomega.Eventually(ctx, func(ctx context.Context) (*resourcev1alpha2.ResourceClaim, error) { return b.f.ClientSet.ResourceV1alpha2().ResourceClaims(b.f.Namespace.Name).Get(ctx, claim.Name, metav1.GetOptions{}) }).WithTimeout(f.Timeouts.PodDelete).Should(gomega.HaveField("Status.Allocation", (*resourcev1alpha2.AllocationResult)(nil))) }) </ins> }) ginkgo.Context("multiple nodes", func() {

When a ResourceClaim: uses delayed allocation does not get created for a specific pod (i.e. not from a template) then it currently gets allocated and remains allocated until: scheduling of a pod fails because the node originally chosen for the allocation isn't suitable, in which case the scheduler triggers deallocation via the user deletes the claim What we want instead is that deallocation gets triggered when the last user of such an allocated claim gets removed from . To avoid races, deallocation must be done through . This can be implemented either as a controller or directly in the API server.
/assign
/triage accepted /sig node /priority important-longterm /kind feature


Question: } fakeDocker.containerList = []docker.APIContainers{ { <del> Names: []string{"foo--qux--1234"}, </del> <ins> Names: []string{"/k8s--foo--qux--1234"}, </ins> }, { <del> Names: []string{"bar--qux--1234"}, </del> <ins> Names: []string{"/k8s--bar--qux--1234"}, </ins> }, } fakeDocker.container = &docker.Container{

release-master-blocking Conformance-GCE-master-kubetest2Kubernetes e2e suite.[It] [sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance] ci-kubernetes-gce-conformance-latest-kubetest2.Overall kubetest2.Test 07-11 15:51 EEST No response /sig cloud-provider cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I remember some timeout has changed with ginkgo2 and something moved from 24hours to 1 hours, or something like that, is it possible that this is the problem here, in this job?
The Grinko timeout change could also be the root problem with this job:
hmm, we have to revisit the flags
hmm, this seems correct
that was a good observations, seems that job is failing because gathering the logs takes too much time !
there is no timeout set in the kubetest2 command here (not sure if this would be the place to set a timeout)
it seems kubetest has to be migrated to ginkgov2 it should use this as basis I'm on vacation this week, can somebody take over and open the issue and patch kubetest2?
looks like this file needs to get updated: add line 72:
oh, you already commented this :D
I will have a look later
Related: umbrella issue:
last testgrid run passed :)


Question: framework.ExpectNoError(err) e2eservice.WaitForServiceUpdatedWithFinalizer(cs, svc.Namespace, svc.Name, true) }) <ins> ginkgo.It("should find a service from listing all namespaces", func() { ginkgo.By("fetching services") svcs, _ := f.ClientSet.CoreV1().Services("").List(metav1.ListOptions{}) foundSvc := false for _, svc := range svcs.Items { if svc.ObjectMeta.Name == "kubernetes" && svc.ObjectMeta.Namespace == "default" { foundSvc = true break } } framework.ExpectEqual(foundSvc, true, "could not find service 'kubernetes' in service list in all namespaces") }) </ins> }) // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed.

This issue proposes the writing of a new test for the API endpoint with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. There are no tests listing services, see the following: This mock test uses client-go's functions and hits the endpoints. This shows the endpoints hit by my approach: /sig architecture /area conformance


Question: "//pkg/api/legacyscheme:go_default_library", "//pkg/cloudprovider/providers/gce:go_default_library", "//pkg/scheduler:go_default_library", <ins> "//pkg/scheduler/algorithmprovider/defaults:go_default_library", </ins> "//pkg/scheduler/api:go_default_library", "//pkg/scheduler/factory:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library",

What happened: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Maybe this error is introduced in by one commit (I haven't yet found it). And we need init DefaultProvider when we setup Scheduler in . /sig scheduling /kind bug /assign /cc
This is an interesting issue! I also see it on my machine, but as you pointed out, there is no recent change that could possibly cause this. In fact, we were in code freeze and no PR got merged into the schedule code base in the past ~couple of weeks. I wonder if a change in our build tools are made recently that introduced this.


Question: Items []Foo `json:"items"` } <del> var _ = Describe("ThirdParty resources", func() { </del> <ins> // This test is marked flaky pending namespace controller observing dynamic creation of new third party types. var _ = Describe("ThirdParty resources [Flaky] [Disruptive]", func() { </ins> f := framework.NewDefaultFramework("thirdparty")

ThirdPartyResources (TPR) E2E test creates a custom resource type, and creates an instance of that type in a namespace. The namespace is purged, but the namespace controller DOES NOT observe the dynamic mutation of the type hierarchy. As a result, the instances of that type are never purged, and we could potentially leak resources. In prior discussions of we agreed that users of TPR would have to kick the namespace controller in the interim in order for it to observe the creation of the new TPR type and have it take part in namespace lifecycle. If TPR is a priority, we should address the namespace controller issue in kube 1.5, and eliminate the [Flaky] flag from the TPR E2E. The updates in the e2e framework that intropsect the server dynamically to verify all resources are deleted are getting tripped up by the presence of this test and impacting all others. Here is an example: https://k8s- /cc
+1 to [Flaky] - this is really causing a ton of flakes
Also note that this doesn't just affect the TPR e2e suite - the existence of the TPR can cause any other e2e suite to fail.
/cc

https://k8s- Failed: ThirdParty resources Simple Third Party creating/deleting thirdparty objects works [Conformance] {Kubernetes e2e suite}
cc
Seems like the namespace creation/deletion process is flaky. IIRC there have been some recent PRs touching this area, is that right?
We recently code to try to expose the real underlying error for why the ns deletion fails. This is hopefully now showing it.
The issue is with TPR not integrated properly with the namespace controller. Creation of a new type requires a restart of the controller-manager given current state of the code. The test must be marked flaky until the state of the art is improved (which is not happening in 1.4).


Question: cmd.Flags().StringVar(&o.SortBy, "sort-by", o.SortBy, "If non-empty, sort nodes list using specified field. The field can be either 'cpu' or 'memory'.") cmd.Flags().BoolVar(&o.NoHeaders, "no-headers", o.NoHeaders, "If present, print output without headers") cmd.Flags().BoolVar(&o.UseProtocolBuffers, "use-protocol-buffers", o.UseProtocolBuffers, "Enables using protocol-buffers to access Metrics API.") <ins> cmd.Flags().BoolVar(&o.ShowCapacity, "show-capacity", o.ShowCapacity, "Print node resources based on Capacity instead of Allocatable(default) of the nodes.") </ins> return cmd }

What happened: We ran the following, and got the following: How/why do we have a node reporting 100% memory usage? (There seems to be plenty of memory on the host as reported by the kernel's statistic. (multiple gigabytes) What you expected to happen: Memory usage can't exceed 100%, no? How to reproduce it (as minimally and precisely as possible): we unfortunately don't know Anything else we need to know?: no swap on these VMs. We're curious what kernel memory statistic goes into computing the total from Kubernetes; it's my understanding that there is various ways to go over 100%, e.g., by summing RSS over several processes. (E.g., shared and resident pages would get double-counted.) Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Azure OS (e.g: ): AKS instance? Kernel (e.g. ): (unsure, since this is Azure AKS; we don't have good access to this piece of data…) Install tools: Azure? Network plugin and version (if this is a network-related bug): N/A / Azure Others: None
(Taking my best shot here, but if this is wrong please freely adjust it.) /sig instrumentation
Other than 101% reporting issue, do you observe any other abnormality ?
Well, yes, the node didn't seem to be actually at 100% memory use; as mentioned, it seemed to have significant headroom.
how about
Node Memory utilization is ratio of node Working Set Bytes and node allocatable memory. Allocatable memory is available on node object: Node Working Set Bytes are available on api: Please provide results from those commands so we can distinquish if problem is in kubectl top or in metrics pipeline.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have similar problem: kubectl describe node secondary-pool-dqiqzzikb5 kubectl get --raw I have no idea what's causing it or how to fix :/
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen We have the same issue.
Reopened this issue. <detailsIn response to : /reopen We have the same issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /triage accepted /assign
Seeing same issue here too
Another data point (on AWS EKS): Seems like happen only on smaller nodetypes (I see it with both micro/nano, others are medium) Usage is displayed correctly with NodeMetrics:
Why kubelet node memory utilization can exceed 100% (same can happen for CPU)? Simple, because it's not utilization of physical device, but utilization of resources allocated for pods and system daemons. How utilization of resources allocated is calculated? Simple, it's sum of resources used by on node divided by all resources allocated. How Kubelet allocates resources for pods? It takes all resources available on VM and substracts resources that it reserves for itself, kernel etc. How Kubelet knows how much resources it needs to reserve? User provides it in flags.

There is no error, there is no Kubernetes magic that gets us over 100%, it's just question of how you define utilization. Kubelet reserves some resources for system, they are not included when calculating node utilization, which means it can go above 100% when pods start using reserved resources (this is what over-committing means in "kubectl describe node" output). Some math based on your kubectl describe node output Node capacity: 968424Ki Node allocatable: 559848Ki Node Usage 717664Ki Node utilization: Node usage / node allocatable = 717664Ki / 559848Ki = 128% VM utilization: Node usage / node capacity = 717664Ki / 968424Ki = 74%
Based on my reading of the , I don't think "Node Usage" include only usage from pods. The includes usage by system daemons, so I'm not sure it makes sense to compare to allocatable, which is meant to exclude system daemons.
Hmm, I was not aware of that. This is a good point, as utilization was implemented in kubectl by different SIG (CLI), it's possible that it was not properly reviewed by other stakeholders (SIG Instrumentation/SIG Node). I would make sense to revisit what should be displayed as node utilization.


Question: # Execute after running tests to perform any required clean-up function test-teardown { <del> echo "Vagrant ignores tear-down" 1>&2 </del> <ins> kube-down </ins> } # Set the {user} and {password} environment values required to interact with provider

We should call kube-down in e2e when running teardown. Currently we just . In doing so we also need to ensure that minion-2 is torn down. When calling cluster/kube- manually master and minion-1 are torn down correctly while minion-2 is missed. /cc
I think sorting this out should resolve any issues from and


Question: return true, nil, nil } <del> // haveOverlap searches two arrays and returns true if they have at least one common element; returns false otherwise. func haveOverlap(a1, a2 []string) bool { if len(a1) > len(a2) { a1, a2 = a2, a1 } m := map[string]bool{} for _, val := range a1 { m[val] = true } for _, val := range a2 { if _, ok := m[val]; ok { return true } } return false } </del> // GeneralPredicates checks a group of predicates that the kubelet cares about. // DEPRECATED: this exist only because kubelet uses it. We should change kubelet to execute the individual predicates it requires. func GeneralPredicates(pod *v1.Pod, meta Metadata, nodeInfo *schedulernodeinfo.NodeInfo) (bool, []PredicateFailureReason, error) {

/sig scheduling /priority important-soon See as an example. Part of /assign
GitHub didn't allow me to assign the following users: angao. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig scheduling /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thx, I will do it soon.


Question: gateEnabled: false, want:        nil, }, <ins> "enabled with valid spec but nil APIGroup": { spec:        validSpecNilAPIGroup, gateEnabled: true, want:        validSpecNilAPIGroup.DataSource, }, "disabled with valid spec but nil APIGroup": { spec:        validSpecNilAPIGroup, gateEnabled: false, want:        nil, }, </ins> } for testName, test := range tests {

What happened: While implementing the new VolumePVCDataSource feature I discovered that even though the we don't use APIGroup (the appropriate setting is ) if the APIGroup is omitted it results in a nil pointer exception on the API server. What you expected to happen: APIGroup should be able to be omitted from the spec How to reproduce it (as minimally and precisely as possible): Create a spec to create a clone of a PVC with out the APIGroup parameter in the DataSource spec: The result will be an nil pointer exception during the verification of the datasource field. Adding the APIGroup entry will address this as a work-around: Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-alpha.1.+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2019-06-12T22:12:02Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Install tools: local-up-
I have a simple fix, will post PR in a moment
/sig storage
/priority critical-urgent /milestone v1.15


Question: go_test( name = "go_default_test", <del> srcs = ["convert_test.go"], </del> <ins> srcs = [ "convert_test.go", "import_known_versions_test.go", ], </ins> data = [ "//test/fixtures", ], embed = [":go_default_library"], deps = [ <ins> "//pkg/api/legacyscheme:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", </ins> "//staging/src/k8s.io/cli-runtime/pkg/genericclioptions:go_default_library", <ins> "//staging/src/k8s.io/client-go/kubernetes/scheme:go_default_library", </ins> "//staging/src/k8s.io/client-go/rest/fake:go_default_library", "//staging/src/k8s.io/kubectl/pkg/cmd/testing:go_default_library", ],

What happened: Tried to convert a v1beta1 Ingress object to v1, and received the following error: What you expected to happen: The conversion to work. How to reproduce it (as minimally and precisely as possible): Then run: /sig cli /assign
Can i work on this issue?
/assign
/triage accepted


Question: t.Errorf("Expected error when calling GetMetrics on incorrectly initialized MetricsBlock, actual nil") } } <del> func TestGetMetricsBlock(t *testing.T) { // FIXME: this test is Linux specific if runtime.GOOS == "windows" { t.Skip("Block device detection is Linux specific, no Windows support") } // find a block device // get all available block devices //  - ls /sys/block devices, err := os.ReadDir("/dev") if err != nil { t.Skipf("Could not read devices from /dev: %v", err) } else if len(devices) == 0 { t.Skip("No devices found") } // for each device, check if it is available in /dev devNode := "" var stat fs.FileInfo for _, device := range devices { // if the device exists, use it, return devNode = "/dev/" + device.Name() stat, err = os.Stat(devNode) if err == nil { if stat.Mode().Type() == fs.ModeDevice { break } } // set to an empty string, so we can do validation of the last // device too devNode = "" } // if no devices are found, or none exists in /dev, skip this part if devNode == "" { t.Skip("Could not find a block device under /dev") } // when we get here, devNode points to an existing block device metrics := NewMetricsBlock(devNode) actual, err := metrics.GetMetrics() if err != nil { t.Errorf("Unexpected error when calling GetMetrics: %v", err) } if a := actual.Capacity.Value(); a <= 0 { t.Errorf("Expected Capacity %d to be greater than 0.", a) } } </del>

On a clean Ubuntu dev environment I get the following error when I executed the I think that the tests should not fail and should handle the case where there is a cdrom device but there is not an actual CD media inserted. I got feedback from and this is also happing on his environment with the following error: I am coping also his comment from the slack that may help: Looks like it's not a problem with cdrom, but some dependence on device. I don't think it should try try to access at all. This is my very first issue, I am new to this, any help is welcome. I am really looking forward to starting coding but I am overwhelmed with the amount of information that I need to digest. /cc serathius /sig instrumentation
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Fact that this tests fails only locally and not on CI means that there will not be a lot of interest in fixing it. Feel free to prepare a fix for it. Please let me know if you need help with setting up development workflow.
this is probably related to Because other tests have this same issue, CI is still running unit tests as UID=0. I have work to remove that but I wasn't sure about another storage test.
The block metrics tests were introduced in - I am fine with deleting the test in question. We have e2e that captures this more accurately and unit test seems kinda less useful anyways.
opened to remove the test. I think we should be fine without it.
Thank you! Also just for the record noting that I can of course locally reproduce this (with in my case).
thanks all. similar issues are tracked in ,


Question: petMounts := []api.VolumeMount{{Name: "datadir", MountPath: "/data/"}} podMounts := []api.VolumeMount{{Name: "home", MountPath: "/home"}} ps := newStatefulSet(psName, ns, headlessSvcName, 3, petMounts, podMounts, labels) <ins> setInitializedAnnotation(ps, "false") </ins> _, err := c.Apps().StatefulSets(ns).Create(ps) Expect(err).NotTo(HaveOccurred())

What happened: What you expected to happen: The statefulset should have scaled down to 0. Instead, even after 10 minutes: How to reproduce it (as minimally and precisely as possible): Run the e2e test on a gce cluster running the latest and greatest kubernetes. cc
if the petset has replicas=0 it's not going to do anything
When I a running petset, and set its scale to 0, that seems to work and delete the pets, and we should have a functioning scaler. Am I misunderstanding something? I'm using the same teardown process for the e2e test I linked in the issue as the other tests (), but it times out sometimes after network partition.
yeah is scale in the petset controller noticing deletion timestamp? because the test code you showed doesn't care
As I mentioned yesterday, it isn't even issuing a delete request even after 10 minutes, because the pods continue to be "running". I'll check the controller manager logs today. I suspect it might be the backoff in the rate limited work-queue.
Hmm, the AGE fields on your pets don't make sense if it isn't even issuing a delete?
In the output I pasted above, at the 7 minute mark, it created the new replacement pet-0, after the network partition was healed. Then tried to scale down the StatefulSet and waited 10 minutes for the pods to disappear.
I see a bunch of different failures modes on the bug: 3 08:58:44.475: Failed waiting for pods to enter running: timed out waiting for the condition Failed to scale statefulset to 0 in 10m0s. Remaining pods: [] 3 08:11:30.728: Unexpected error: <nilBasically 3 translates to We should clarify. 1 and 2 both have events like guessing the node controlelr shouldn't be setting timestamp?
Looking into the case 2: The controller manager log goes: So I guess even deletions are being blocked by the health/readiness.
Both issues appear to be due to the testsuite and the way we use the "initialized" annotation in tests. I'll send a PR to fix these shortly. Thanks


Question: - file: /etc/init.d/kubelet {% endif %} - file: /var/lib/kubelet/kubernetes_auth <ins> {% if pillar.get('enable_node_monitoring', '').lower() == 'true' %} - file: /etc/kubernetes/manifests/cadvisor.manifest {% endif %} </ins>

On latest pull, the Vagrant kube- scripts completes successfully, but the Salt setup in the minion keeps going forever. End result is a cluster that is not useable. Looking at '/var/log/salt/minion' on the minion box, see lots of entries like: 2015-03-04 20:09:11,952 [ ][INFO ] Starting a new job with PID 2015-03-04 20:09:11,954 [ ][INFO ] Returning information for job: 2015-03-04 20:09:12,027 [ ][DEBUG ] Decrypting the current master AES key 2015-03-04 20:09:12,028 [ ][DEBUG ] Loaded minion key: 2015-03-04 20:09:12,217 [ ][INFO ] User vagrant Executing command with jid 2015-03-04 20:09:12,217 [ ][DEBUG ] Command details {'tgt_type': 'glob', 'jid': '', 'tgt': '*', 'ret': '', 'user': 'vagrant', 'arg': [], 'fun': ''} No other help looking at the master's salt log. This is Vagrant 1.7.2, with VirtualBox 4.3.20 on Mac OSX (Yosemite).
Is this repeatable for you if kube-down and kube-up?
Yes, this is reproducible on my laptop. But, I was able to get it up by doing a kube-push after the failed kube-up. , Eric Tune wrote:
Self assigning and will try to reproduce. Sent from my iPhone
I was able to reproduce. It appears that the Kubelet fails to start properly on first start:
The push basically caused the Kubelet to restart, and then it does work. Without the push, systemd would have eventually restarted the Kubelet. We need more investigation as to what was causing the Kubelet to fail on initial startup.


Question: }) }) }) <ins> var _ = SIGDescribe("[Feature:Windows] Kubelet-Stats", func() { f := framework.NewDefaultFramework("kubelet-stats-test-windows") ginkgo.Describe("Kubelet stats collection for Windows nodes", func() { ginkgo.Context("when running 3 pods", func() { // 10 seconds is the default scrape timeout for metrics-server and kube-prometheus ginkgo.It("should return within 10 seconds", func() { ginkgo.By("Selecting a Windows node") targetNode, err := findWindowsNode(f) framework.ExpectNoError(err, "Error finding Windows node") framework.Logf("Using node: %v", targetNode.Name) ginkgo.By("Scheduling 3 pods") powershellImage := imageutils.GetConfig(imageutils.BusyBox) pods := newKubeletStatsTestPods(3, powershellImage, targetNode.Name) f.PodClient().CreateBatch(pods) ginkgo.By("Waiting up to 3 minutes for pods to be running") timeout := 3 * time.Minute e2epod.WaitForPodsRunningReady(f.ClientSet, f.Namespace.Name, 3, 0, timeout, make(map[string]string)) ginkgo.By("Getting kubelet stats 1 time") iterations := 1 var totalDurationMs int64 for i := 0; i < iterations; i++ { start := time.Now() nodeStats, err := e2ekubelet.GetStatsSummary(f.ClientSet, targetNode.Name) duration := time.Since(start) totalDurationMs += duration.Milliseconds() framework.ExpectNoError(err, "Error getting kubelet stats") // Perform some basic sanity checks on retrieved stats for pods in this test's namespace statsChecked := 0 for _, podStats := range nodeStats.Pods { if podStats.PodRef.Namespace != f.Namespace.Name { continue } statsChecked = statsChecked + 1 framework.ExpectEqual(*podStats.CPU.UsageCoreNanoSeconds > 0, true, "Pod stats should not report 0 cpu usage") framework.ExpectEqual(*podStats.Memory.WorkingSetBytes > 0, true, "Pod stats should not report 0 bytes for memory working set ") for _, containerStats := range podStats.Containers { framework.ExpectEqual(containerStats.Logs != nil, true, "Pod stats should have container log stats") framework.ExpectEqual(*containerStats.Logs.AvailableBytes > 0, true, "container log stats should not report 0 bytes for AvailableBytes") } } framework.ExpectEqual(statsChecked, 3, "Should find stats for 10 pods in kubelet stats") time.Sleep(5 * time.Second) } avgDurationMs := totalDurationMs / int64(iterations) durationMatch := avgDurationMs <= time.Duration(10*time.Second).Milliseconds() framework.Logf("Getting kubelet stats for node %v took an average of %v milliseconds over %v iterations", targetNode.Name, avgDurationMs, iterations) framework.ExpectEqual(durationMatch, true, "Collecting kubelet stats should not take longer than 10 seconds") }) }) }) }) </ins> // findWindowsNode finds a Windows node that is Ready and Schedulable func findWindowsNode(f *framework.Framework) (v1.Node, error) {

Lets put a metrics CI job in place that isnt . Investigate our test coverage of metrics-server See why our CI jobs dont test metrics-server Confirm that we are doing enough in is a good test, lets make it part of all our containerd jobs by removing the Serial tags
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
while were at it make that iterations a non-magic number
while were at it make sure the metrics stats are non-zero , so that we catch regressions turns out we DO run SERIAL jobs in the sig-windows dashboard either we dont collect enough metrics, so the tests pass, or we do, and we dont watch the dashboards :) https://k8s- <-- check the sig-windows dashboard...
this test starts 10 pods on a node at once and run into issues starting 10 pods on a single node when this was run in parallel with other jobs. Maybe we should consider adding a different metrics job that just starts 1-3 pods and does deeper validation of the metrics retruned?
Hey, is this issue still open? I would like to work on this, I am new to open source
We cannot de-serialize the existing metrics job due to its nature. We can however create new test cases that do not need to be run in parallel and do deeper validation of metrics like Jay suggested above. I'm not sure if is still working on the issue. If not feel free to work on this. Thanks!

/unassign
/retitle Add non-serialized metrics tests for Windows
do we have to make an another var _ with [Serial] not included in it and run 3 pods instead of 10 in it. Help needed, am I thinking correct here.


Question: etcd_over_ssl: 'false' EOF fi <ins> if [ -n "${ETCD_QUORUM_READ:-}" ]; then cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls etcd_quorum_read: '$(echo "${ETCD_QUORUM_READ}" | sed -e "s/'/''/g")' EOF fi </ins> # Configuration changes for test clusters if [ -n "${APISERVER_TEST_ARGS:-}" ]; then cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls

Slack user "nalin" reported an issue Looking at the code this makes sense. Will fix.
For anyone following along, the fix for this bug unfortunately didn't get cherrypicked because of a process issue, and is broken until 1.5.2. cc
I just the new lines from As a temperatur fix ONLY. If anyone like me dont want to wait. Worked like at charm for AWS cluster setup. :)
A quick note, just as confirmation: the above temporary fix does resolve the issue for AWS - even though it's a change to a GCE-related file (cluster/gce/configure-). It seems that the GCE files are used for certain aws functions.
In based AWS deployment, the script is shared between AWS and GCE.
Also, this fix was cherry-picked into the branch a while ago, so instead of manually adding the lines, you can also just checkout the branch and run from there.


Question: etcd_over_ssl: 'false' EOF fi <ins> if [ -n "${ETCD_QUORUM_READ:-}" ]; then cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls etcd_quorum_read: '$(echo "${ETCD_QUORUM_READ}" | sed -e "s/'/''/g")' EOF fi </ins> # Configuration changes for test clusters if [ -n "${APISERVER_TEST_ARGS:-}" ]; then cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls

Slack user "nalin" reported an issue Looking at the code this makes sense. Will fix.
For anyone following along, the fix for this bug unfortunately didn't get cherrypicked because of a process issue, and is broken until 1.5.2. cc
I just the new lines from As a temperatur fix ONLY. If anyone like me dont want to wait. Worked like at charm for AWS cluster setup. :)
A quick note, just as confirmation: the above temporary fix does resolve the issue for AWS - even though it's a change to a GCE-related file (cluster/gce/configure-). It seems that the GCE files are used for certain aws functions.
In based AWS deployment, the script is shared between AWS and GCE.
Also, this fix was cherry-picked into the branch a while ago, so instead of manually adding the lines, you can also just checkout the branch and run from there.


Question: } func (a *azureDiskAttacher) WaitForAttach(spec *volume.Spec, devicePath string, _ *v1.Pod, timeout time.Duration) (string, error) { <del> var err error </del> volumeSource, _, err := getVolumeSource(spec) if err != nil { return "", err

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There is a drawback after PR: following code will fetch vmCache may lead to below error and the vmCache refresh time period would be 1 min, and may cost 1 min which is too long time How to fix this At least in Linux, we may still use parameter to get LUN, which is almost zero time. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
k8s 1.10.4, Azure (deployed via ACS-engine) do you know about any directly applicable workaround? This is hitting us quite hard. logs: logs: The last Attach operation reports success, yet we enter the same error loop over and over again.
thanks for reporting, I am fixing this issue, code should be ready in this week.
We have seen what looks like this issue - possible regression? It took 11m for the volume to attach.
it's more likely that disk could not be detected under dir inside vmss instance, is it a transient issue? Can this issue repro easily?
Seems to happen consistently on the 2nd pod of a new deploy. We have seen 3-4 times in a row deleting and re-deploying this project. Doesn't seem to be an issue for the kafka-0 and kafka-2 pods, just kafka-1. And only on a fresh deploy of the helm chart, deleting the pod does not reproduce.
what's your region and k8s version? it only happened on vmss, right?
The version we’re running is k8s 1.13.12 in useast2. And right this is on vmss. EDIT: I should also note that we're also using multi fault zones for each of the vmss pools we're deploying.
I'm having a similar issue, but I'm on 1.14.8.

I don't know if is related, but these disk attachments seem to not be working properly on aks with vmms.
confirm volume attach with vmss is failing bad..
pls open a separate issue, provide detailed error info, e.g. k8s version, pod error description, etc. thanks.
This took too long AKS - kubernetes version -1.14.8 location -west europe
thanks for the reporting, I could confirm that the is due to PR: , which is going to mitigate vmss API throttling issue, in that PR, it enlarge the getVMSS TTL from 1min to 10min. Fortunately, from k8s 1.15.0, won't have this issue, since we have refactored the disk detect logic, it won't depend on getVMSS operation to get the disk LUN. In brief, pls upgrade to latest version 1.15.5, 1.15.x won't have this issue, thanks.
cherry-pick PR: to fix this issue on 1.14
You mention a 1.14 cherry pick but I'm also seeing Is this correct to assume 1.13 will have this fix as well ?
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! --/kind bug Uncomment only one, leave it on its own line: /kind bug /kind feature What happened: There is a drawback after PR: following code will fetch vmCache may lead to below error and the vmCache refresh time period would be 1 min, and may cost 1 min which is too long time At least in Linux, we may still use parameter to get LUN, which is almost zero time. Anything else we need to know?: Kubernetes version (use ): v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
1.13 don't accept any cherry-pick, so only 1.14 could be cherry-picked now, target fixed version is 1.14.10
the delay still can be very big this is because azure api call for disk attach returns after at 35s-2min and kubelet goes to its exp. backoff retries so it can reach 2min timeout adding attached recording: called 3 disk simultaneous attach with interval ~2s and saw and So actual attachment takes ~10-30s (device appears in /dev/disk/azure/scsi1) , but due to azure api we can loop on it I saw also that disk attach timing is unstable and somtemes it can take 2.5min , what do you think about logic like below? is not waiting for attachdisk to return, instead it calls GET until it returns lun id (we saw it takes few seconds after attachdisk). after GET returns lun id , it updates node's disks with lunId and kubelet can continue with mount after waiting for /dev/disk/azure/scsi1/lun*
same here
It takes about 10 minutes to recover from the FailedMount messages.
what's your k8s version?
there is a new vhd disk feature based on azure file which could attach/detach disk < 1s, consider that as an option if you have concern about disk attach/detach time cost:
AKS v1.14.8
for this specific issue, aks 1.15.x won't have this issue.
hi ,I am not dare to upgrade.because my stateful rabbitmq with pvc can not started because of failmount of PVC!
you could scale your statefulset to 0 replica, and pvc disk would be detached. After upgrade, scale back to original count.
okay,let me have a try.
I use helm to can I restore to two replicas after upgrading k8s version?


Question: match: registry.k8s.io/build-image/debian-base:[a-zA-Z]+-v((([0-9]+).([0-9]+).([0-9]+)(?:-([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?) - name: "registry.k8s.io/distroless-iptables: dependents" <del> version: v0.6.1 </del> <ins> version: v0.5.6 </ins> refPaths: - path: build/common.sh match: __default_distroless_iptables_version=

sgtm, add an issue labeled as release-blocking and in the milestone to either promote to 1.23.0 or revert this to 1.22.x before our 1.31 release Originally posted by in Kubernetes is being built and tested with go1.23rc2 at HEAD () to let it bake in the CI and help iron out issues, if any, with this version of Go. However, for the 1.31 release, the version of Go should be a non-rc, released version. The release timeline of go1.23.0 is ~same as the release timeline of K8s 1.31.0 (see ). If go1.23.0 is not out by then, we should revert to go1.22.x before cutting K8s 1.31.0. /sig release architecture /area code-organization /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /cc
/milestone v1.31
cc
xposting relevant slack thread:
Also unlike in the past, missing shipping 1.23.0 in 1.31.0 is not a big deal, we have much more room to switch to 1.23.0 in a patch release after all the compat changes with the go team. So we should not fear reverting if the timelines don't line up.
(though go1.23 adds the thing that makes it workable to default to old go version compat settings, so we'll have to eke out another 5 months on 1.21/1.22 for until we can adopt 1.23 on release branches in ~Jan if we miss 1.31, but that's how it looks like it will have to happen)


Question: dbMetricsMonitorJitter = 0.5 ) <ins> // TODO(negz): Stop using a package scoped logger. At the time of writing we're // creating an etcd client for each CRD. We need to pass each etcd client a // logger or each client will create its own, which comes with a significant // memory cost (around 20% of the API server's memory when hundreds of CRDs are // present). The correct fix here is to not create a client per CRD. See // https://github.com/kubernetes/kubernetes/issues/111476 for more. var etcd3ClientLogger *zap.Logger </ins> func init() { // grpcprom auto-registers (via an init function) their client metrics, since we are opting out of // using the global prometheus registry and using our own wrapped global registry,

The project uses a lot of CRDs - in some cases almost 2,000. I do mean here, not custom resources. We've noticed that API server memory consumption balloons when many CRDs are installed - for example with ~1,900 CRDs we see the API server using up to ~8Gi RSS. A little profiling shows that around 1.5Gi of that memory is being used by the etcd3 client's logger. With Kubernetes built from with the CRDs at loaded I'm seeing the API server using up to ~7.5GiB RSS. I see it dip to ~5.5GiB, presumably when garbage collection runs. You can see taking up a ton of memory below. Zap is the logger used by the etcd client. Note that this is a sample, so it's actually using more than the indicated ~800MiB. ! I'm seeing the API server using up to ~6GiB RSS. I see it dip to ~4.5GiB, presumably due to GC runs. You can see no longer appears below. ! I would not expect a logger to be responsible for such a significant amount of the API server's memory consumption. You can get the above profile by: out . to build a kind node. to start a control plane for the above build. to allow anonymous pprof access the CRDs from I used to see the RSS but I imagine would show it too. See for details on how I disabled the logger to get the "without logger" numbers. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
Since you looked at the code, and I'd rather not spend time doing that. Any chance you can tell if it's because it's being mis-used?
This is a scale dimension where the apiserver has historically . With the OpenAPI discovery performance being improved it looks like this might be a hint at what the next bottleneck might be? Any idea how many etcd clients were instantiated? I suspect it's quite a lot.
Sadly I didn't yet look deeply past turning it off. I was slightly optimistic that we might just be able to disable it without any meaningful impact.

What I do know is that the offending code seems to be on the etcd3 client end - i.e. it defaults to creating its own logger when one isn't supplied. You can see that code at
I don't - will try to find out.
The zap logs might tell you if you have the data. If my memory of the code is right, you should get 1 etcd client per CRD right now (plus some for builtin types).
/triage accepted
Based on my read of the code, you're right. I see roughly: All of /apis is handled by a crdHandler When a request comes in it gets an instance of the crd (from an informer) Gets serving info from a map of CRDs - if there is none it... Calls customresource.NewStorage which creates a genericregistry.Store Calls its CompleteWithOptions method That calls its “Decorator”, which is a genericregistry.StorageWithCacher That calls generic.NewRawStorage which just wraps factory.Create factory.Create calls newETCD3Storage which calls newETCD3Client I'm experiencing technical difficulties getting lsof running in my kind container but /proc/pid/net confirms we have a connection to etcd per CRD: Note that 1,937 - 60 = 1,877 - the number of CRDs we have loaded (the 1,878 above includes the header line).


Question: } } <del> if err := checkPermissions(); err != nil { klog.Error(err) } </del> utilruntime.ReallyCrash = s.ReallyCrashForTesting // TODO(vmarmol): Do this through container config.

What happened: follow on to Need to comment and avoid checking UID on windows. requires some research into what we should be checking so that we can add appropriate TODO. What you expected to happen: Lets also check this at the call site, Anything else we need to know?: Environment: Kubernetes version (use ): master OS (e.g: ): windows
/assign
/assign
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
Hi are you working on this issue already? If not I'd like to tackle this issue :)
Hi yeah I am already working on this :)
Started branch for this here
thanks
Hey guys is there anything I can do here?
If you want to test perks patch that would be great ?
That sounds great Where do I get started?
Hey ! is there anything I can help with? and how?
I dont think there is anything left to do anything more that you think we need to do to get this merged?
Tested this with containerd as well, I think its good to test
/triage accepted


Question: e2essh "k8s.io/kubernetes/test/e2e/framework/ssh" ) <del> // TestContext should be used by all tests to access common context data. var TestContext framework.TestContextType </del> // New local storage types to support local storage capacity isolation var localStorageCapacityIsolation featuregate.Feature = "LocalStorageCapacityIsolation"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Flexvolume e2e tests are skipped in https://k8s-, so as far as I can tell we don't have Flexvolume test coverage. They are skipped because they require SSH access to drop the Flexvolume plugin onto the host, but these test clusters do not have SSH access. We should change all Flexvolume tests to use the DaemonSet installation mechanism instead which does not require SSH access. See and Flexvolume tests I'm aware of are below, but there could be more: Looking for help with this bug! /sig storage /priority important-soon /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --What happened: Flexvolume e2e tests are skipped in https://k8s-, so as far as I can tell we don't have Flexvolume test coverage. /priority important-soon /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, Can I work on this issue? I think i know how to handle it.
are you still working on this? I'm interested in helping out as well.
/assign


Question: linux/s390x linux/ppc64le darwin/amd64 <ins> darwin/arm64 </ins> windows/amd64 windows/386 )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When trying to build kubectl from tip () on a M1 Macbook: Why is this needed: M1 MacBooks are a new major client platform that many developers will use. brew listed kubectl (kubernetes-cli) is one of the major packages still not supported on M1 () Since it's supported by go, it's only required to add it to the supported client list Locally I managed to add support using the following patch:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli architecture
/sig release
Any updates on this? cheers
you can watch the PR:
it works, nice!!!
Done!!!


Question: } cgroupConfig.Resources.SkipDevices = true <ins> // if the hugetlb controller is missing supportedControllers := getSupportedUnifiedControllers() if !supportedControllers.Has("hugetlb") { cgroupConfig.Resources.HugetlbLimit = nil // the cgroup is not present, but its not required so skip it klog.V(6).Infof("Optional subsystem not supported: hugetlb") } </ins> manager, err := cgroupfs2.NewManager(cgroupConfig, cgroupConfig.Path, false) if err != nil { return fmt.Errorf("failed to create cgroup v2 manager: %v", err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet doesn't start up on a cgroup2 host without hugetlb controller What you expected to happen: Should work without hugetlb controller How to reproduce it (as minimally and precisely as possible): Boot Ubuntu 20.04 host with . Make sure cgroup v2 controllers are enabled but hugetlb is not: Install Moby from its master branch (Docker 20.0X). Binaries are available at Install kind from its master branch (kubernetes-sigs/kind Run with kubernetes/kubernetes Run Run Environment
cc
/sig sig-node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-node </details
/sig node
/area kubelet
looks like you already saw since you filed :)
That’s for containerd, not for kubelet :)
right :)
we can relax the hugetlb requirement, since it is also optional on cgroup v1. could you please try if the following patch is enough to fix the issue on your environment?
LOL this again :-) guess this isn't just glinux with hugetlb issues :upsidedownface:
IKR!!!
Perfect, could you open a PR? Now kind works on cgroup v2!
opened a PR here:


Question: t.Errorf("[%v]Unexpected test result. Expected delete %v, got %v", description, expectDelete, podDeleted) } } <ins> // TestPodDeletionEvent Verify that the output events are as expected func TestPodDeletionEvent(t *testing.T) { f := func(path cmp.Path) bool { switch path.String() { // These fields change at runtime, so ignore it case "LastTimestamp", "FirstTimestamp", "ObjectMeta.Name": return true } return false } t.Run("emitPodDeletionEvent", func(t *testing.T) { controller := &NoExecuteTaintManager{} recorder := testutil.NewFakeRecorder() controller.recorder = recorder controller.emitPodDeletionEvent(types.NamespacedName{ Name:      "test", Namespace: "test", }) want := []*v1.Event{ { ObjectMeta: metav1.ObjectMeta{ Namespace: "test", }, InvolvedObject: v1.ObjectReference{ Kind:       "Pod", APIVersion: "v1", Namespace:  "test", Name:       "test", }, Reason:  "TaintManagerEviction", Type:    "Normal", Count:   1, Message: "Marking for deletion Pod test/test", Source:  v1.EventSource{Component: "nodeControllerTest"}, }, } if diff := cmp.Diff(want, recorder.Events, cmp.FilterPath(f, cmp.Ignore())); len(diff) > 0 { t.Errorf("emitPodDeletionEvent() returned data (-want,+got):n%s", diff) } }) t.Run("emitCancelPodDeletionEvent", func(t *testing.T) { controller := &NoExecuteTaintManager{} recorder := testutil.NewFakeRecorder() controller.recorder = recorder controller.emitCancelPodDeletionEvent(types.NamespacedName{ Name:      "test", Namespace: "test", }) want := []*v1.Event{ { ObjectMeta: metav1.ObjectMeta{ Namespace: "test", }, InvolvedObject: v1.ObjectReference{ Kind:       "Pod", APIVersion: "v1", Namespace:  "test", Name:       "test", }, Reason:  "TaintManagerEviction", Type:    "Normal", Count:   1, Message: "Cancelling deletion of Pod test/test", Source:  v1.EventSource{Component: "nodeControllerTest"}, }, } if diff := cmp.Diff(want, recorder.Events, cmp.FilterPath(f, cmp.Ignore())); len(diff) > 0 { t.Errorf("emitPodDeletionEvent() returned data (-want,+got):n%s", diff) } }) } </ins>

Events from node taint manager doesn't report involved object apiVersion: ! apiVersion is reported as other events. For example: ! a pod on node A (not master) off node A is reported, but without apiVersion in involvedObject. The apiVersion is not reported in The issue is still not fixed in master: ! <details</details<detailsIrrelevant </details<details</details<details</details<details</details<details</details
/sig node /sig apps
/assign In the latest version, the issue is confirmed
one suggestion you should wait for the corresponding SIG to triage the issue and accept it before sending PRs, the chances of your PR getting reviewed and approver are much higher
You are right sorry about that. It won’t happen again.
no no worries, it was a suggestion, is not that you did something wrong, but it is possible that your PR doesn't get reviews and you get frustrated
/remove-sig node please re-add if this is node-related issue
I think it is a node related issue. The root cause in taint manager.
/sig node
/assign /triage accepted


Question: - -k - --cache-size=1000 - --no-negcache <ins> - --dns-loop-detect </ins> - --log-facility=- - --server=/__PILLAR__DNS__DOMAIN__/127.0.0.1#10053 - --server=/in-addr.arpa/127.0.0.1#10053

Is this a BUG REPORT or FEATURE REQUEST?: What happened: If hosts points to a local DNS server, or if it contains the cluster's DNS server (to enable looking up services from host), it'll cause DNS loop which can effectively disable kube-dns or make it very slow. What you expected to happen: At least, kube-dns should detect localhost or cluster DNS server entries in and not send queries to them. How to reproduce it (as minimally and precisely as possible): Use a system with local cashing DNS server, or add cluster's DNS server address to hosts Anything else we need to know?: According to documentation, adding should make dnsmasq detect that and disable the upstream DNS server which cause the loop. It should prevent DNS from becoming unresponsive in the first case (a better solution is to extract real upstream servers, which probably cannot be done for every configuration); and for the second case it should skip sending queries to itself and send to real upstream servers.
/sig network
Could you describe the use-case more? If the host's contains cluster DNS server and some other upstream server - like 8.8.8.8, will services lookups work reliably from the host? If the services lookup goes to the 8.8.8.8 server, it will return NXDOMAIN. If it hits the clusterDNS server, only then it would work, correct?
Also ref If the first server returns NXDOMAIN, the second server will not be tried. How do you make both cluster DNS name and non-cluster DNS name resolvable on the host?
If cluster DNS server comes before a real DNS server, it works fine. Also note that what man page says is the glibc (or libc) behavior, not other users (e.g. dnsmasq sends its queries to any one of them, but can be configured to query them in strict order ( flag), or to query all of them (). Anyway, consider when cluster DNS server comes before the real server: cluster DNS queries, the cluster DNS server responds with the IP of the cluster or pod. external names, the query is sent to cluster DNS server. It doesn't find the name, so it tries to send it to upstream DNS servers; which in my example are cluster DNS server itself and an external DNS server.

If dnsmasq sends it to external DNS server, it'll receive the answer and can serve the query. If it sends it to itself, there is a loop and the same process happens again until dnsmasq decides to query the other server (which might not happen!). WIth flag, when dnsmasq sends the query to itself, it'll recognize the loop and will remove its own address from the list of upstream DNS servers, so it'll always send external names to the external DNS server from now on.
Thanks for the explanation. I believe it might work with explicit configuration. Though adding a flag to upstream for fulfilling a not quite common use case seems a bit odd to me. Have you thought of configuring upstream DNS servers via kube-dns config [1] such that it will not send request to itself at all? [1] -
Note that the flag has two use cases, the other one is prevent kube-dns breakage when upstream DNS server is a local one (e.g. If you deploy to Ubuntu, which has local caching DNS server by default (e.g. in 18.04, it is systemd-resolved and contains 127.0.0.53)). Actually I guess there are a number of DNS related issues which might be solved by this flag too. So, while this flag will not make kubernetes magically find real upstream dns servers in such cases, it prevents kube-dns crash; which has caused frustration and some time for me to discover. So, either kubelet should error out and exit if the given contains a local address, or such flag should be used by kube-dns to prevent DNS loops. And, IIRC, the only reason that cluster's DNS server is not to host, at least by kubespray, is mentioned to be host specific in the docs. So, my use case seems to be an ideal one and it would be the default one if it was easy to setup for different platforms. About using kube-dns config, no. We're going to provide automatic kubernetes deployment for many environments, and so we don't know the upstream DNS servers upfront, and we don't want to hard code it. We want it to be the DNS servers used in the target network. So, configuring a specific upstream DNS server is a no-go.


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/types" </ins> "github.com/golang/glog" )

now shows NodePhase, NodeCondition, etc. information from NodeStatus. It should also show related events like what we did for pods.


Question: }, expected: []*batchv1.Job{}, }, <ins> { name: "test getting jobs whose labels do not match job template", cronJob: &batchv1.CronJob{ ObjectMeta: metav1.ObjectMeta{Namespace: "foo-ns", Name: "fooer"}, Spec: batchv1.CronJobSpec{JobTemplate: batchv1.JobTemplateSpec{ ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{"key": "value"}}, }}, }, jobs: []runtime.Object{ &batchv1.Job{ObjectMeta: metav1.ObjectMeta{ Namespace:       "foo-ns", Name:            "foo-fooer-owner-ref", Labels:          map[string]string{"key": "different-value"}, OwnerReferences: []metav1.OwnerReference{{Name: "fooer", Controller: &trueRef}}}, }, &batchv1.Job{ObjectMeta: metav1.ObjectMeta{ Namespace:       "foo-ns", Name:            "foo-other-owner-ref", Labels:          map[string]string{"key": "different-value"}, OwnerReferences: []metav1.OwnerReference{{Name: "another-cronjob", Controller: &trueRef}}}, }, }, expected: []*batchv1.Job{{ ObjectMeta: metav1.ObjectMeta{ Namespace:       "foo-ns", Name:            "foo-fooer-owner-ref", Labels:          map[string]string{"key": "different-value"}, OwnerReferences: []metav1.OwnerReference{{Name: "fooer", Controller: &trueRef}}}, }}, }, </ins> } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) {

Changing the label on a job template in a CronJob causes any Jobs held by the successful/failed history parameters to stop being tracked, so they never get cleaned up. Jobs with labels that don't match the (upgraded) CronJob, but still have an owner reference, should be subject to the history limit. a CronJob with a job template containing label , and non-zero history limit the CronJob to run and generate some Jobs (default 3 successful, 1 failed) the CronJob with a job template containing label the CronJob to run and generate some more Jobs there are more than 3 successful Jobs/1 failed Job! I believe this is the relevant code: The CronJob controller finds Jobs that it should care about by matching on the labels in the job template, this obviously breaks if the labels in the job template change. 21 Self-hosted No response No response No response No response_
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
Experiencing another side-effect of this bug. If job template labels are updated while a cronjob has status (a scheduled job is running), then that cronjob gets stuck in . The cause seems to be the same as describes: once the job template labels are updated, cronjob controller V2 loses track of the jobs with the old labels, and therefore never updates the status of that active cronjob to indicate that the scheduled job has completed.
/assign


Question: // queue. If pod is unschedulable, it adds pod to unschedulable queue if // p.receivedMoveRequest is false or to backoff queue if p.receivedMoveRequest // is true but pod is subject to backoff. In other cases, it adds pod to active <del> // queue. </del> <ins> // queue and clears p.receivedMoveRequest. </ins> func (p *PriorityQueue) AddUnschedulableIfNotPresent(pod *v1.Pod) error { p.lock.Lock() defer p.lock.Unlock()

What happened: Kubernetes scheduler may leave a single pod stuck in pending state after a worker node is reloaded/reinstalled. This problem appears to have been introduced in Kubernetes v1.12.4 and v1.13.1. We have not been able to recreate the problem in our testing on earlier releases. The key to the failure is that the worker being reloaded/reinstalled must be the only scheduling option available. The pending pod is eventually scheduled if a new scheduler leader is elected or the node status changes. This appears to be a timing bug since the problem does not always occur. What you expected to happen: No pods are stuck in pending state. How to reproduce it (as minimally and precisely as possible): Start with a single node cluster running a handful of pods. Cordon, drain and then reload/reinstall the node. Since this is a timing bug, it may take several iterations to recreate the problem. Environment: Kubernetes version (use ): 1.12.4, 1.13.1 and 1.13.2 Cloud provider or hardware configuration: VMs running in VirtualBox OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Custom ansible scripts Others: N/A
/sig scheduling
Events from the pending pod: Events from the node:
does "reload/reinstall" mean info is still kept in etcd? or (or equivalent steps) is executed to remove that node entirely? And I assume the "reloaded" node keeps the same spec and IP/node name with previous one?
the info is still kept in etcd. And the node keeps the same spec, IP and name as it had before. The reload does the following: and drain the node the VM the VM again with the same spec the node
What's the taint/toleration accordingly?
there were no taints to the node during this process. The pods are part of a deployment with only basic tolerations provided by Kubernetes. Deleting the pod will get it scheduled even though the taints and tolerations are unchanged. Simply restarting the scheduler leader will get the pod scheduled as well.
Can you share and ?
The symptom (esp. "a single pod stuck in pending state") is quite similar as issue .

I think why this occurs since v1.12.4 is related the PR - which was delivered to master branch, and backported to v1.12.4, and it appears to reveal higher chance of racing. cc/ Basically we have 2 solutions to fix this: the real root cause of the racing condition and give fix (I haven't got luck to reproduce this yet) that we have a solution () to alleviate this kind of problem, could you try to run your test on a) master branch or b) v1.12.4 with cherrypicked to see if the issue is resolved? and does this fits your business need? (in other words, the single pod may be pending for 30 seconds at most)
I'll recreate and provide you the details. However, the pod that is stuck pending is different every time.
Thanks, I'll work on the recreate and test of the cherrypick. That PR certainly seems promising.
Node YAML after recreate:
Pod YAML after recreate:
the cherrypick doesn't look trivial for 1.12 or 1.13. Give me some time to setup an environment to test on 1.14. Unless you have another way?
Yeap, it's not a trivial task to rebuild the source code. Let me know if you need any help. I suppose you're going to try on latest 1.14 tag? e.g. v1.14.0-alpha.1? Right now I'm working on the 1st solution I mentioned in The challenge is I can't reproduce this in my local dev env (I'm running on source code branch of v1.12.4) yet, so my current plan is to retry reproducing your issue to make my solution more convincing. If still no luck, I will send out the PR for reviewing early next week.
And let me confirm with your statement: When the issue occurs, everytime there is only one single pod stuck in pending state, right?
Still don't have luck in reproducing the issue. But I've prepared a PR and put detailed analysis there. Any comments are welcome.
No, the problem does not occur every time. It can take as many as 7 retries before reproducing it.

Sry, I should have made it more clear - I wanted to confirm that among all the times the issue is reproducible, is that everytime there is only one single pod in pending state?
correct, every time that I've reproduced the problem there has been only one pod in pending state.
Thanks. To some extent, it's consistent with the deduction in my PR.
is it possible to have the necessary fixes cherry-picked to at least 1.12 and 1.13? That would make it easier for me to build and test?
Yeap, cherrypick PRs ( onto 1.13, onto 1.12) were created and will be reviewed/approved soon.
Thank you for the cherrypick PRs. I rebuilt 1.13 with the cherrypick PR and still saw the problem on 2 of 10 recreate attempts. Without the fix, I saw a similar failure rate.
the fix is only ready in master right now. Did you rebuild 1.13 by manually running “git cherrypick <this commit id” on release-1.13? Or directly built from latest src code of release-1.13?
I manually rebuilt 1.13.2 with
/reopen I will take another look.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I've verified is able to fix your issue thoroughly. And cherrypick PRs have been raised (1.12 , 1.13 ).
Thank you.
Thanks, and K8s 13.3 already has another change () that should alleviate the problem significantly. K8s 13.4 will have the definitive fix. Given that the root-cause is fixed, we can close this issue.


Question: group *APIGroup } <ins> const ( StatusUnprocessableEntity = 422 ) </ins> // Handle returns a Handler function that expose the provided storage interfaces // as RESTful resources at prefix, serialized by codec, and also includes the support // http resources.

I0825 04:43:42. ] GET /logs: (51.111us) 301 goroutine 315 [running]: (respLogger).WriteHeader(0xc21036a240, 0x12d) +0x9c net/http.Redirect(0x7f4d60b7ed80, 0xc21036a240, 0xc2101a7680, 0xc2102bbd60, 0x6, ...) +0x436 net/http.(redirectHandler).ServeHTTP(0xc2103089c0, 0x7f4d60b7ed80, 0xc21036a240, 0xc2101a7680) +0x5e net/http.(ServeMux).ServeHTTP(0xc2102f4030, 0x7f4d60b7ed80, 0xc21036a240, 0xc2101a7680) +0x163 ·002(0x7f4d60b7ed48, 0xc2102c5140, 0xc2101a7680) +0x197 net/http.HandlerFunc.ServeHTTP(0xc210309180, 0x7f4d60b7ed48, 0xc2102c5140, 0xc2101a7680) +0x40 (defaultAPIServer).ServeHTTP(0xc210308a60, 0x7f4d60b7ed48, 0xc2102c5140, 0xc2101a7680) +0x51 net/http.serverHandler.ServeHTTP(0xc210317960, 0x7f4d60b7ed48, 0xc2102c5140, 0xc2101a7680) +0x16e net/http.(conn).serve(0xc2102b2180) +0x7b7 created by net/http.(Server).Serve +0x28b
Is it just that 301 isn't in the list of things to not show a stacktrace for?


Question: tokens[privilegedLoopbackToken] = &user.DefaultInfo{ Name:   user.APIServerUser, UID:    uuid.New().String(), <del> Groups: []string{user.SystemPrivilegedGroup}, </del> <ins> Groups: []string{user.SystemPrivilegedGroup, user.AllAuthenticated}, } tokens[UnprivilegedUserToken] = &user.DefaultInfo{ Name:   "unprivileged", UID:    uuid.New().String(), Groups: []string{user.AllAuthenticated}, </ins> } tokenAuthenticator := authenticatorfactory.NewFromTokens(tokens, controlPlaneConfig.GenericConfig.Authentication.APIAudiences) if controlPlaneConfig.GenericConfig.Authentication.Authenticator == nil { <del> controlPlaneConfig.GenericConfig.Authentication.Authenticator = authenticatorunion.New(tokenAuthenticator, authauthenticator.RequestFunc(alwaysEmpty)) </del> <ins> controlPlaneConfig.GenericConfig.Authentication.Authenticator = authenticatorunion.New(tokenAuthenticator, authauthenticator.RequestFunc(unsecuredUser)) </ins> } else { controlPlaneConfig.GenericConfig.Authentication.Authenticator = authenticatorunion.New(tokenAuthenticator, controlPlaneConfig.GenericConfig.Authentication.Authenticator) }

/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake <!-- Please assign a SIG using: /sig SIG-NAME --/sig api-machinery
/assign I want to take a try
2 but actual 0 till timeout resource services, expected 2, actual 0 error waiting or ResourceQuota status: timed out waiting for the condition time later, actual state became 2 Seems polls a ResourceQuota status with timeout 10s may be not enough, we should enlarge it into 300s, that is poll for 300s timeout instead of 10s.
/triage accepted


Question: //nolint:unused // U1000 see comment above, we know it's unused normally. func (fakeTestDeps) SnapshotCoverage() {} <ins> //nolint:unused // U1000 see comment above, we know it's unused normally. func (fakeTestDeps) InitRuntimeCoverage() (mode string, tearDown func(string, string) (string, error), snapcov func() float64) { return "", nil, nil } </ins>

Hello k8s folks: This code here: is making a call to the function , which is documented as not maintaining the Go 1 compatibility guarantee. Here is the header comment for MainStart: Filing this issue to let you know that in the upcoming Go release (1.23) the signature of MainStart is indeed changing; there is a new method on the testDeps interface. If you could please add this new method to your dummy testDeps interface (as in the previous patch that would be great. Thanks. from the golang team. N/A N/A No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
Closing this issue. <detailsIn response to : a better place to ask would be on the support channels. please see: /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen </details
/sig release
/assign mauri870
/triage accepted
/sig testing
many thanks for the report. release folks, let's start the go1.23 tracking issue we can link this to as a PR that will need picking to release branches to make them ready for eventual update to 1.23
cc


Question: func (e *eventBroadcasterImpl) StartStructuredLogging(verbosity klog.Level) watch.Interface { return e.StartEventWatcher( func(e *v1.Event) { <del> klog.V(verbosity).InfoS("Event occurred", "object", klog.KRef(e.InvolvedObject.Namespace, e.InvolvedObject.Name), "kind", e.InvolvedObject.Kind, "apiVersion", e.InvolvedObject.APIVersion, "type", e.Type, "reason", e.Reason, "message", e.Message) </del> <ins> klog.V(verbosity).InfoS("Event occurred", "object", klog.KRef(e.InvolvedObject.Namespace, e.InvolvedObject.Name), "fieldPath", e.InvolvedObject.FieldPath, "kind", e.InvolvedObject.Kind, "apiVersion", e.InvolvedObject.APIVersion, "type", e.Type, "reason", e.Reason, "message", e.Message) </ins> }) }

I used to use 1.171 standalone Kubelet mode without an API server for my cluster. Because I’m not using API server, I can only get container events from Kubelet logs. But after I upgrade Kubelet to 1.20.1, I found that the log format is changed and Kubelet log is no longer logging container info(“FieldPath” field in logs) for events logs due to this change So with a newer version Kubelet, I cannot know which container those events belong to from Kubelet logs. container info(“FieldPath” field) should be back to Kubelet event logs Can reproduce it with any standalone Kubelet version after 1.18.20. I also want to what’s the best practice to get events for standalone Kubelet? Should “FieldPath” field add back to log? or is there any plan to add something like an “event server”, so users can configure and use it to get events for standalone Kubelet easier? 20.1 N/A <details</details<details</details<details</details<details</details
/sig node
seems to only change the log style. In this example, the pod name is still included in the log. /cc /sig instrumentation
/wg structured-logging
Pod name is still there, but the "FieldPath" field which includes container name is missing Old version log:
To add back FieldPath, change to An example of an event for pod: The log included name/namespace/kind/apiVersion/
/triage accepted
would you confirm this is expected or a regression?
/assign
Gentle ping on this, this is a blocking issue, without a fix, we need to stay with v1.18. Do we have a plan to fix this? or maybe I can help to fix it. Thanks
Removing fields like "fieldPath" was not intended. It should be ok to reintroduce this field to logs like showed in This is however not a bug as we don't provide any guarantee about log contents. Feel free to send a PR that will add this field back. /kind feature
/remove-kind bug


Question: "hybrid_network.go", "memory_limits.go", "networking.go", <ins> "security_context.go", </ins> "service.go", "volumes.go", ],

We need test cases that cover running as Windows users in containers. Windows uses a different field WindowsContainerSecurityContext.runasusername () They should only run with the ginkgo focus set. This will replace the similar test cases on Linux: /kind test /sig windows
/good-first-issue
/assign
GitHub didn't allow me to assign the following users: michellemcone. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I can't assign this to Michelle yet, so I'll park it on myself for now. If it's not done in a few weeks and someone else wants to take it, drop a comment here.
/milestone v1.14
/remove good-first-issue It looks like this may not be ready for testing. has the first half of the feature implemented, but it's still incomplete.
/assign So, when the API for `? Although this come at a later date, since it would need to be implemented in kubelet first.
GitHub didn't allow me to assign the following users: bclau. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign </details
/good-first-issue remove
/clear milestone
/milestone clear /kind cleanup
/remove-good-first-issue
/milestone v1.16
Hello! I'm part of the bug triage team for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16? If not please re-tag the issue to the planned milestone.
This issue is addressed by: This should be included in 1.16.


Question: sha256 = "05ab3937bc68562e989dc143362ec4d4275262ba9f359338aed720fc914457a5", url = "https://storage.googleapis.com/kubernetes-release/network-plugins/cni-amd64-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz", ) <ins> docker_pull( name = "debian-iptables-amd64", digest = "sha256:adde513f7b3561042cd2d2af4d2d355189bbb2f579584b5766e7d07be4f7e71e",  # v7 registry = "gcr.io", repository = "google-containers/debian-iptables-amd64", ) </ins>

BUG REPORT Kubernetes version 2 Environment: Install tools: What happened: The code in takes care to supply the flag so it will wait on a lock to avoid interference between multiple processes modifying the table. If you read the source you discover it does this by creating a file . However, when is running in a container, is in a local filesystem, so it is not able to lock against processes running on the host or in other containers. And before you decide that the answer is to mount from the host, look at all the issues around leaking netns, e.g. Credit for pointing this out to me. Credit for pointing out locking functionality was to a few months back
Ironically, this is caused by a bugfix that moved to file-based locking. Previously, iptables would lock by reserving an abstract unix socket, which is tied to the network namespace.
Some discussion also in and following comments.
I experimented with this: but it didn't work on CoreOS, as the host wasn't using a version of iptables that used the file lock yet. Just ended up creating a directory call
I guess I should file this as an issue against kubeadm and any of the other sources of kube-proxy container config.
/assign
Should we leave this open (waiting for ... ?)
Waiting for I guess... (which I'm reviewing now)
i think this should not be marked as complete until the kubeadm fixes are in.
/reopen
/close fix in


Question: logReduction: logreduction.NewLogReduction(identicalErrorDelay), } <del> if err := service.determineAPIVersion(conn); err != nil { </del> <ins> if err := service.determineAPIVersion(conn, endpoint); err != nil { </ins> return nil, err }

When installing the latest component, got the error: Then the flag , got error: It is confusing to see these error msgs at the first glance, need to make it more clearly. Add the required field validation, and add more supportive error message. Download the latest v1.25.0 binary, and install it with systemd service or just manually start the binary, will reproduce these errors. <details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
kubelet 1.24+ is not working with the docker, unless using Are you using it? If not, try switching to other available container runtimes. Please re-open if you have further questions /close
Closing this issue. <detailsIn response to : kubelet 1.24+ is not working with the docker, unless using Are you using it? If not, try switching to other available container runtimes. Please re-open if you have further questions Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, I got to know the , so need to add runtime validation by the fix PR: PTAL thanks.


Question: # See the OWNERS docs at https://go.k8s.io/owners <del> maintainers: - adityadani </del> <ins> approvers: - lpabon - saad-ali - rootfs reviewers: - lpabon - saad-ali - rootfs </ins>

Found this when trying to see which vendored dependencies does not use a proper tag (and is using a random SHA). The dependency is very old and there have been a lot of updates to it: Looks like the However there does not seem to be anyone around to take care of the portworx plugin. Is it time to deprecate it? is anyone using it? /sig storage
cc ?
/area code-organization
Hello The portworx volume plugin is definitely in use. The portworx volume plugin doesn't need frequent updates since the plugin acts as a simple client to all provisioning calls. In the same view, the vendored openstorage package doesn't need to be updated too. The server side implementation in Portworx handles all the latest Portworx features.
cc:
in that case, can you please update the dependency to a specific tag instead of a random SHA?
/assign updated the title of the issue as well
my request is not for a new release or tag in libopenstorage/openstorage repository. Just to update the SHA we use in our k/k repository to one of the released versions ( ). This is not asking or demanding anything to be done in libopenstorage/openstorage, just that we should update what we use from there (and avoid a random SHA). We are trying to do this for all vendored libraries (one by one … slowly as part of the sig-arch code organization sub project, so it’s not a random request)
/assign
GitHub didn't allow me to assign the following users: lpabon. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After a chat with - apparently this request is coming across the wrong way I am closing this out as it seems to add a over head /close
Closing this issue. <detailsIn response to : After a chat with - apparently this request is coming across the wrong way /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: genericfeatures.APIListChunking: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.32 <del> genericfeatures.APIPriorityAndFairness: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.31 </del> genericfeatures.APIResponseCompression: {Default: true, PreRelease: featuregate.Beta}, genericfeatures.APIServerIdentity: {Default: true, PreRelease: featuregate.Beta},

Priority and Fairness is going GA in 1.29, the feature has been marked stable and the feature gate has been locked to default. See In 1.31, we need to remove the feature gate.
/kind bug /sig api-machinery /cc
/assign
/triage accepted


Question: Force           bool ServerSideApply bool ForceConflicts  bool <ins> genericclioptions.IOStreams </ins> } var _ Object = &InfoObject{}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I tried to with a Deployment spec that was roughly shaped like this (note that the is invalid, since it should be a list of instead of ). This caused kubectl to fail to create a three-way patch and panic. <details<summaryClick here for the stacktrace</summary</detailsWhat you expected to happen: An error is returned How to reproduce it (as minimally and precisely as possible): Create a deployment, then try to diff using the YAML provided above (make sure to replace the name and namespace) Anything else we need to know?: The panic occurs on this line because is nil: It's set to nil here: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig cli
Raised a pr which may fix this bug.


Question: container "google.golang.org/api/container/v1" "google.golang.org/api/googleapi" "google.golang.org/cloud/compute/metadata" <del> "gopkg.in/gcfg.v1" </del> ) const (

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.3.317+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-01T05:48:36Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.3.317+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-01T05:48:36Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64"} What happened: Dynamic provisioned volumes in different zones with master failed to deleted What you expected to happen: Since the volume can be dynamic provisioned when the zones differs from master, then it should be deleted after delete the PVC which created them. How to reproduce it (as minimally and precisely as possible): up an all-in-one cluster. The instance is created in zone us-central1-a. several storageClass. One in zone us-central1-a. Another in zone us-central1-b. several PVC. pvc1 using storageClass1(zone us-central1-a), pvc2 using storageClass2(zone us-central1-b). PV/volume were provisioned in proper size/mode/zone. PVCs. with volume in zone us-central1-b failed with error can not find the volume.

Anything else do we need to know: Name: pvc--70d1-11e6-b17e- Labels: failure- failure- Status: Failed Claim: default/pvc5 Reclaim Policy: Delete Access Modes: RWO Capacity: 4Gi Message: Delete of volume "pvc--70d1-11e6-b17e-" failed: GCE persistent disk not found: "kubernetes-dynamic-pvc--70d1-11e6-b17e-" Source: Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine) PDName: kubernetes-dynamic-pvc--70d1-11e6-b17e- FSType: Partition: 0 ReadOnly: false Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message 11m 11m 1 {persistentvolume-controller } Warning VolumeFailedDelete Delete of volume "pvc--70d1-11e6-b17e-" failed: GCE persistent disk not found: "kubernetes-dynamic-pvc--70d1-11e6-b17e-" creationTimestamp: '2016-09-01T22:49:26.439-07:00' description: '{"":"pvc--70d1-11e6-b17e-","":"pvc5","":"default"}' id: '' kind: compute#disk name: kubernetes-dynamic-pvc--70d1-11e6-b17e- selfLink: sizeGb: '4' status: READY type: zone:
Kubernetes can delete GCE PDs only in zones it manages. I.e. either in the same zone as the master or, when multizone is enabled in GCE config, in all zones in the same region. This should not be allowed in the first place. Kubernetes should not create disks outside zones it manages and report an error instead. So... please use following , it will be allow creation/deletion of disks in all zones in us-central1 region: And I'll fix Kubernetes to report an error when creating a disk in all other zones.


Question: }, }, options: printers.GenerateOptions{Wide: true}, <del> // Columns: Name, Completions, Duration, Age, Containers, Images, Selectors </del> <ins> // Columns: Name, Status, Completions, Duration, Age, Containers, Images, Selectors </ins> expected: []metav1.TableRow{ { <del> Cells: []interface{}{"job1", "1/2", "", "0s", "fake-job-container1,fake-job-container2", "fake-job-image1,fake-job-image2", "job-label=job-label-value"}, </del> <ins> Cells: []interface{}{"job1", "Running", "1/2", "", "0s", "fake-job-container1,fake-job-container2", "fake-job-image1,fake-job-image2", "job-label=job-label-value"}, </ins> }, }, },

Suspend is a GA feature and it is big component of Kueue. It would be nice to display if jobs are suspended when doing . A new column that displays if the job is suspended would be helpful. Improve UX around suspend. Its a bit difficult to know why the job has not created pods unless you know about the suspend field.
/sig cli /wg batch /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig cli /wg batch /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Just something I noticed as I was testing some job in JobSet. It would be nice to see which jobs are suspended when doing . Hope its not a controversial change.
cc who worked on
Not controversial at all IMO /triage accepted
/assign
Any suggestions on where to place this column? I'm thinking between completions and durations. Or, should this be displayed only when the output is set to wide?
I would put it in the default view as well. Probably before completions
Addressed in the PR:


Question: assert.Equal(t, secret1.Namespace, updatedSecret.Namespace) assert.True(t, secretsEqual(secret1, *updatedSecret), fmt.Sprintf("expected: %v, actual: %v", secret1, *updatedSecret)) <ins> // Wait for the secret to be updated in the informer store. err = WaitForSecretStoreUpdate( secretController.secretFederatedInformer.GetTargetStore(), cluster1.Name, types.NamespacedName{Namespace: secret1.Namespace, Name: secret1.Name}.String(), updatedSecret, wait.ForeverTestTimeout) assert.Nil(t, err, "secret should have been updated in the informer store") </ins> // Test update federated secret. secret1.Data = map[string][]byte{

https://k8s- Failed: TestSecretController {secret} Previous issues for this test:
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
This flake seems to have started after merged (just looking at the merge time line). Can you please take a look?
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
Sorry just saw this, looking at it now
Sent for a fix
nice thanks


Question: prometheus.MustRegister(E2eSchedulingLatency) prometheus.MustRegister(SchedulingAlgorithmLatency) prometheus.MustRegister(BindingLatency) <ins> prometheus.MustRegister(SchedulingAlgorithmPredicateEvaluationDuration) prometheus.MustRegister(SchedulingAlgorithmPriorityEvaluationDuration) </ins> }) }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --As of now in , we have 3 metrics, for end2endLatency, algorithmLatency and BindingLatency. The problem with algorithmLatency is it is a combination of predicate and priority latencies. So breaking down this metric into predicate and priority latencies would be really helpful for performance analysis. For example, as part of the analysis after these metrics were , we were able to find latencies when we have taints & tolerations alone. In fact, ability to drill down on every predicate and priority would be ideal but want to know other's opinions on this.
/cc
+1 please send a PR.
We also have some demonstrations of how this helps with things like highlighting taint and toleration performance and so on. We should update readme info on how to use these along side the scheduler perf stuff as we've done internally .
You need to be careful about what latency means if we are computing multiple different predicates in parallel (if we decide to split it by predicate). Splitting predicate vs priority is definitely a good idea.
Please label this as part of our 1.7 scheduling forensics milestone .
Let's remove the idea of latency entirely, it is confusing - this is just an attempt at measuring predicate and priority timescales, and there is no response latency that we care about in those small (20ms) timescales , (with the exception of maybe the latency of an etcd hit that might happen in a client if the cache is missing information I guess)
Not really - there were cases of predicates which computation could take over a second.

Yes, and even longer on large clusters with things light affinity.... However here I'm just disputing use of the term latency since latency is usually used to refer to the time between the end of a request and the beggining of a response. What we endeavor to do here is really measure round trip execution time, for predicates, in isolation. If we want to measure some kind of latency then I believe we would need to define some notion of a request/response, which has diminished value outside of something like RPC. If we truly were measuring latency (and it might be interesting to do so with tims tracing work) then we might thus look at the time between writing a new unscheduled pod to the system, and the time it takes to start running the scheduler fit functions or something... Tl;dr there are a lot of varying usages and definitions for latency, but almost all of them, with the exception of measurements of virtual function calls, refer to some unavoidable minimum cost that occurs during distributed communication, and in this case, we are simply measuring "execution time of the predicate filter" which is called from inside the same process. (Alternatively I guess it's ok to call it "fitfunctionpredicate_latency", I guess, since then we are using the fit function as the context for a response measurement).
OK, IIUC you object usage of the word 'latency', right? We use it in this context for quite some time, and as I agree that it doesn't fit formal definition, there's no point in changing it now, as it would confuse people even more.
yeah, i really think they should all be refactored to not be named as such, that is my broader point . but if we want to keep those metrics id suggest we at least update the definition in the fields to clarify that latency is defined WRT rather then to a RPC or some other typical latency calc scenario. aplogies for the bike shed, ill move this to a PR :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: // Restore from the checkpoint path // NOTE: This MUST happen before creating the apiserver source // below, or the checkpoint would override the source of truth. <del> updatechannel := cfg.Channel(kubetypes.ApiserverSource) </del> <ins> var updatechannel chan<- interface{} </ins> if bootstrapCheckpointPath != "" { glog.Infof("Adding checkpoint path: %v", bootstrapCheckpointPath) <ins> updatechannel = cfg.Channel(kubetypes.ApiserverSource) </ins> err := cfg.Restore(bootstrapCheckpointPath, updatechannel) if err != nil { return nil, err

/kind bug cc. What happened: Since - specifically , kubelet always calls , leading it to always register an apiserver source with the . This leads to to never return true in cases where the kubelet is not configured to communicate with an apiserver. The direct result of this is that when removing a manifest from the directory, kubelet never kills the containers. What you expected to happen: The containers get killed properly. How to reproduce it (as minimally and precisely as possible): Configure kubelet standalone without an apiserver, and with a static pod manifest path. Add a manifest, then remove it. Observe the containers stick around. Set logging to greater than level 6, and see a lot of . Environment: Kubernetes version (use ): 1.9.2 (1.8.7 works)
- Is there a run a static manifest that's not a master test case in the node-e2e suite?
I don't think there are standalone kubelet tests since it's hard to observe the pod status.


Question: } pvcCopy.Status.Capacity = capacity <ins> pvcCopy.Status.Conditions = []v1.PersistentVolumeClaimCondition{} </ins> newData, err := json.Marshal(pvcCopy) if err != nil {

We should not attempt resizing of file system if volume is mounted as read-only. /sig storage cc