Question: procGetDiskFreeSpaceEx = modkernel32.NewProc("GetDiskFreeSpaceExW") ) <ins> type UsageInfo struct { Bytes  int64 Inodes int64 } </ins> // Info returns (available bytes, byte capacity, byte usage, total inodes, inodes free, inode usage, error) // for the filesystem that path resides upon. func Info(path string) (int64, int64, int64, int64, int64, int64, error) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: in calculates number of inodes wrong when there are hard links. What you expected to happen: inodes with hardlinks should only be counted once. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Issue is still not solved. There is an open PR waiting for review. /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
still waiting for to be merged

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to run pods on Alpine Linux with busybox find. What you expected to happen: I expected the call to exit with success, not failure. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing GNU findutils, so busybox implementation of is used. Anything else we need to know?: failed because the argument is not implemented in busybox . The option is not specified in so it can not be expected to be supported in all implementations. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related bug Others: I think it would be better to use instead of Related to
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This is solved by which is waiting for a review. /remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Still waiting for the PR to be merged. /remove-lifecycle rotten
/remove-lifecycle rotten

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running test suite on a cluster with workers on Alpine Linux, which uses busybox instead of GNU coreutils fails. The log shows that it fails to execute the command. What you expected to happen: I expected the call to exit with success. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing . Anything else we need to know?: It failed due to the option which is not implemented in busybox. The option is not specified in so I don't think it is reasonable to expect it to implemented in busybox. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related Others:
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale fixes this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten
Still waiting for PR to be merged


Question: hasBasicAuth = true } if len(config.KeystoneURL) > 0 { <del> keystoneAuth, err := newAuthenticatorFromKeystoneURL(config.KeystoneURL) </del> <ins> keystoneAuth, err := newAuthenticatorFromKeystoneURL(config.KeystoneURL, config.KeystoneCAFile) </ins> if err != nil { return nil, nil, err }

Hi, I tried to integrate the keystone auth plugin (which would be perfect for my needs) into the cluster I'm currently setting up. However, I can't get the connection working since our keystone installation is currently only reachable over http As far as I can see in , there is no option to deactivate this check. I think this should be an optional parameter on startup, similar to defaulting to the check being activated.
I don't think we should enable username/password checking over http.
I agree that we are loosing a security layer here, but you can't guarantee you got the password from client over https, so it might be intercepted at this point anyways. More general, I think the proper way to authenticate to Kubernetes with the Keystone plugin using tokens since they will limited lifetime restricted on keystone only which is a big deal regarding keystone likely being bound to a LDAP and thus the password being valid for other services. So I don't want to check username/password directly. I'm just starting out with the Keystone plugin, so I want to point this out in general, if token auth is supported, the better. Couldn't try it in praxis for now since I can't get the auth URL running.
Assigning to to add a priority label.
Setting the priority low until we have a sense of how many users have this restriction (keystone only reachable over http).
I ran into this same issue with trying to deploy kubernetes with devstack in the same vm and wanted to test integration and do some development. I think there are a significant number of developers in this same boat. http is secure enough since it doesn't leave the developers vm and it isn't intended for production. The use case is the same as:
I'd vastly prefer allowing specifying the CA for the keystone server over allowing http auth calls carrying usernames and passwords
I don't disagree. :) But, I also think Kubernetes shouldn't handle the usernames/passwords at all. In http mode, sending a token is much less risky then a username/password since they expire relatively quickly. See this issue for details:
I agree tokens would be preferable, but I wouldn't expect credentials of any kind sent over http to remote servers
Agreed. No one in their right mind should be passing creds or bearer tokens over http over a network.

I'm just interested in being able to easily deploy a k8s in DevStack with a simple plugin, where that k8s is hooked into the deployed keystone so that we can start developing things in OpenStack that depend on k8s. This gives developers a smooth/easy experience to then focus on doing the development stuff and not have to mess with setting up a CA/k8s, which will prevent some of them from contributing to k8s/OpenStack. The idea is simple for developers, secure for operators. Totally separate use cases.
It should be as simple as telling k8s to trust the keystone cert as a CA (assuming the self-signed cert generated by keystone is correctly marked as a CA, which it should be, since it is signing for itself). I've seen too many production installations running with http or untrusted TLS settings to believe that only developers will use this option if it's made available.
I didn't think Devstack could easily deploy with SSL enabled, but it looks like you can: So http support might not be as important as I thought. It could be relegated to a documentation issue for deploying Devstack in this mode when deploying with Kubernetes. We'd still need a way to specify the root CA though for Keystone.
So, I played around with this a little. For my current non devstack based system, I've deployed kubernetes as a cluster running on my laptop deployed as a set of pods via /etc/kubernetes/manifests. Only kubelet is native and all the rest of the services are containers direct from google. Its working well, but since I have 3 api server containers, and get the containers direct from google, I'd have to jump through a bunch of hoops to add the keystone ssl ca cert to each of the containers. I still think an option to specify the ssl cert for keystone would be useful rather then having to merge it into the system's chain of trust.
That's how I'd want to handle this as well


Question: import ( "fmt" <ins> "strings" </ins> "sync" "time"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pods are synchronized by kubelet before network is ready. Synchronization in such case fails and pod is synchronized after 10 seconds. What you expected to happen: Kubelet should wait until network is ready before starting synchronizing pods. How to reproduce it (as minimally and precisely as possible): Configure cluster with Cluster Autoscaler enabled. Start pod so cluster is scaled up. In logs you might find following error: E0907 13:18:49. 972 ] Error syncing pod xxxxxx ("xxxxxxxxx(xxxxxxxxxxx)"), skipping: network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] Environment: Build from HEAD Cloud provider or hardware configuration: GKE
/kind bug /sig node


Question: "strings" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/errors" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/meta" "github.com/GoogleCloudPlatform/kubernetes/pkg/runtime"

e.g. REST.Update should never return an arbitrary error value. Maybe we should change the return code to something stricter than generic error.
/cc
FYI: is for remaining cases where the API doesn't return JSON/Status errors.
Re-opening this until we
I think etcd errors require more abstraction, also:
There are some admission control plugins that did not always follow that practice which I will resolve as part of
Are there any known instances of this remaining?
None that I know of.


Question: expectError:    false, }, { <ins> // case insentive check diskURL:        "/subscriptions/4be8920b-2978-43d7-axyz-04d8549c1d05/resourcegroups/azure-k8s1102/providers/Microsoft.Compute/disks/andy-mghyb1102-dynamic-pvc-f7f014c9-49f4-11e8-ab5c-000d3af7b38e", expectedResult: "azure-k8s1102", expectError:    false, }, { </ins> diskURL:        "/4be8920b-2978-43d7-axyz-04d8549c1d05/resourceGroups/azure-k8s1102/providers/Microsoft.Compute/disks/andy-mghyb1102-dynamic-pvc-f7f014c9-49f4-11e8-ab5c-000d3af7b38e", expectedResult: "", expectError:    true,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: azure resource id should be case insensitve, while some places are case sensitve: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign


Question: return nil, fmt.Errorf("Error getting loadbalancer %s: %v", name, err) } glog.V(2).Infof("Creating loadbalancer %s", name) <del> loadbalancer, err = lbaas.createLoadBalancer(apiService, name) </del> <ins> loadbalancer, err = lbaas.createLoadBalancer(apiService, name, internalAnnotation) </ins> if err != nil { // Unknown error, retry later return nil, fmt.Errorf("Error creating loadbalancer %s: %v", name, err)

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently openstack cloud provider just support for specifying internal LoadBalancerIP. I think that is unreasonable, actually user just use a floating ip to access a LoadBalancer service. So it is necessary to spport for specifying external LoadBalancerIP. What you expected to happen: Support for specifying external LoadBalancerIP on openstack Anything else we need to know?: Currently LoadBalancerIP can be specified in service.Spec.LoadBalancerIP, we can add annotation to distinguish between internal and external.
/sig openstack /assign I'd like to work for it.


Question: var events *v1.EventList // Check for scheduler event about the pod. ginkgo.By("checking for scheduler event about the pod") <del> framework.ExpectNoError(wait.Poll(time.Second*2, time.Second*60, func() (bool, error) { </del> <ins> framework.ExpectNoError(wait.Poll(framework.Poll, 5*time.Minute, func() (bool, error) { </ins> selector := fields.Set{ "involvedObject.kind":      "Pod", "involvedObject.uid":       string(podWithUID.UID),

Appears related to I don't see this flaking in the but it appears to be quite flaky in presubmit CI, where this is the only test on the job that fails. /sig testing node /kind flake /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
/cc
/remove-kind failing-test
I was debugging this test this morning the apiserver logs shows the kubelet events come too late, the test times out, is like they are queued in the kubelet the test fails at 10:24:22 but the event arrives at 10:24:27 but kubelet times doesn't add up, there are vents generated by kubelet at 10:23:07 if we check another job , where the test passes, there is only a few seconds delay in a working job I've analyzed the number of kubelet events on a job with the test passing against failing, and the results shows that the failing test have similar number of events, except the ones with the Reason FailedMount
I could've sworn we'd like banned conformance tests from depending on event delivery because it's not guaranteed to be reliable?
right now the test waits 60 seconds for the event since events are not synchronous, should we have a more reasonable period? EDIT saw Ben's comment after I commented, anyway, the test is failing because it takes more than 1 minutes, at least I couldn't find any failure related to no have any event at all
I'm not familiar with this test, but you're right that Events and Reason aren't things that a Conformance test can depend upon Conformance tests are intended to be stable and backwards compatible according to the standard API deprecation policies. Therefore any test that relies on specific output that is not subject to the deprecation policy cannot be promoted to conformance. Examples of tests which are not eligible to conformance: anything that checks specific Events are generated, as we make no guarantees about the contents of events, nor their deliveryanything that checks optional Condition fields, such as Reason or Message, as these may change over time (however it is reasonable to verify these fields exist or are non-empty)
/milestone v1.23
Did some digging.

It looks like this has been tagged as Conformance since at least 1.2, ref: (sometime prior to 2015-10-07) So this appears to be a case where we missed going back through the existing corpus of conformance tests when we wrote the above doc. is where the rule originated (merged 2018-10-08) I think this test should be demoted from Conformance, since the very nature of what it tests is certain events being generated SIG Node is free to continue to treat this as release-blocking, but it's not something we can reasonable expect of any Kubernetes cluster
Code itself for corresponding test says 1.9: Git history shows the test basically hasn't changed in 1y:
I'll split out demotion into a separate PR, don't want to tangle this issue up
The first single instance of a timeout failure I see is on I'm going to see if I can try to bisect from there, probably merged recently.
to be clear: the test has been around and we should look at fixing it, I was just surprised to see the test name with , we should open a different thread/issue on that topic I am working on the kubernetes deb/RPM releases for the day or else I'd be taking a look.
,%20master%20(dev)%20[non-serial]&width=5 (IPv6),%20master%20(dev)%20%5Bnon-serial%5D&width=5
Opened to cover the conformance demotion
/priority critical-urgent I am proceeding as though we have a perf regression with kubelet events until proven otherwise. Others feel free to also TAL.
I can't reproduce locally on its own, I either will need to stress test it in parallel with some other stuff or I need a slower machine.
/reopen I don't want to just close this because the timeout was bumped if there's a possible regression
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
Staring at notes: Looking at the journal logs I'm seeing some (mild, not unexpected) client-side throttling: So that looks to me like the kubelet thinks it successfully got the event out by 10:23:18 which is well within our 1m timeout. The timestamps in Antonio's post only look at POST, let me see if I can find the PATCH logs from apiserver
From That PATCH was received when the kubelet sent it so the event should have been updated and available... could there be an apiserver latency issue? /sig api-machinery
Possible repro on a just-rebased PR:
I see another PATCH at but I think that's just from the pod getting torn down, it matches the Kubelet logs
this is patching the pod status, probably setting the IPs or something like that, check the URI format is a patch on the pod objetct AFAIK those POST are creating the events , the URI format /api/v1/namespace/<namespace/events ,
latest failure 1:45 am CET fix merged 1:55 AM CET, let's keep it open one day more
I really don't think waiting from 1 -5m is an acceptable fix. This may be catching a performance regression. It has stopped the bleeding but I think the problem remains...
the events , by design, may take 6 mins , so it is completely legit that an event can take 5 mins, you cant' fail if take less (actually, by definition they are unreliable so it should be fine also to not receive it) Performance can not be tested this way, there has to be an explicit test for that, and that test can not use events of course
If the prior "median event lifecycle time" (in the context of this test) was previously < 1 min (and thus test flakes were rare) and now the "median event lifecycle time" has increased so as to trigger regular test flakes, that does seem like something interesting and worthy of prioritization. It just seems probable that there's some "there" there. This could also be explained by environmental changes, did anything change in the underlying test infra that correlates with the increased test failure rate?

We previously had issues where presubmits were failing when periodics were not, see
If the test was newly introduced and was flaking, I'd agree with you, but we've had this test at 100% green since like the 1.2 release, and it only started flaking in the past 48h. This suggests to me that we merged a regression, but since it landed during code freeze, it's very difficult to determine which of the dozens of PRs it was.
Fairly confident there have not been substantial changes to the infra in the past week. For blocking presubmits the infra is moved to public infra owned by the CNCF / SIG K8s infra and the configs are all in shows the CI code being updated regularly, but not the build/test cluster that executes against. KIND upgraded containerd/runc two weeks ago, but that doesn't fit. Given the timing it seems more like:
indeed, check my analysis, a failing job has 429 FailedMount events, a job passing the test only 277, the rest of the events are the same Is not the first time that we have a performance regression or an env problem that causes similar effects, I remember one because of runc, one issue with the DNS on CI machines (that elana linked), other with golang, ... but this is not uncommon at the end of the release when CIs are busier I don't disagree that there can be something, actually I totally agree, I disagree on using this test as a proof or measure of this "something", because: test verifies that, once you create a pod an event from the scheduler and from the kubelet is created. , looking at the code, can take 6 mins and by definition are not realiable , Tests should be clear about what they test , for this cases we need to define a threshold, in sig network we have a test for latency of services, maybe adding one for latency of pods? we can define a threshold there, sig-scalability has it
Latest in triage: It does appear that we had a bunch of CI flakes start showing up as well before increasing the timeouts in the test, so maybe not just a presubmit thing.
/assign
No, it hasn't failed since the timeout bump merged, but the bump suppressing the symptom and hence is making this much more difficult to root cause.

I'd suggest looking at some runs before the timeout was bumped?
Per I'm wondering if some of the timing regressions we're seeing could be due to a perf regression having switched to CRI v1 with containerd. I can't find any corresponding failures on CRI-O. /remove-priority critical-urgent since test is no longer flaking
/unassign I'm not actively working this one
I think this is similar to in that mainly dockershim jobs are affected.
FTR the jobs in the OP are not dockershim based, kubeadm's kinder jobs do often use dockershim but kind (not kinder) does not support dockershim. is looking good though, fwiw
can we close this now?
single 1.21 failure. i'm going to call this closed for now.


Question: podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { <del> nc.maybeDeleteTerminatingPod(obj) </del> pod := obj.(*v1.Pod) if nc.taintManager != nil { nc.taintManager.PodUpdated(nil, pod) } }, UpdateFunc: func(prev, obj interface{}) { <del> nc.maybeDeleteTerminatingPod(obj) </del> prevPod := prev.(*v1.Pod) newPod := obj.(*v1.Pod) if nc.taintManager != nil {

As discussed on slack with we might want to clean the node controller from code that ensures backward compatibility for version 1.4 and earlier ones. /sig node
/assign
- how do we plan to devide this one ? The following order for merge should be fine I think: the tests (we can split also test case deletion if it's too long to review): logic itself + kubeletVersion on the according tests: change how we create nodecontroller: clean consts and BUILD:
Yeah, sounds good.
I'll create the PRs accordingly and assign them to you


Question: # Print the supported API Resources with more information kubectl api-resources -o wide <ins> # Print the supported API Resources sorted by a column kubectl api-resources --sort-by=name </ins> # Print the supported namespaced resources kubectl api-resources --namespaced=true

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a sort parameter (or similar concept) to the command. Why is this needed: With the numerous amount of API resources (not counting custom ones), it is sometimes difficult locating them in the output. By default, API resources are not sorted by the name column (the first on the left) and often this is the column holding the information one wishes to query. A sort parameter would help more easily identify the object to be understood.
/sig usability


Question: docker run --rm -i -v "${TARGET}:${TARGET}:Z" registry.k8s.io/build-image/kube-cross:${KUBE_CROSS_VERSION}  /bin/bash -c " mkdir -p /go/src /go/bin &&  <del> GO111MODULE=on go get -d k8s.io/sample-apiserver@v0.17.0 &&  GO111MODULE=on CGO_ENABLED=0 GOARM=${GOARM} GOOS=${OS} GOARCH=${ARCH} go get k8s.io/sample-apiserver@v0.17.0 &&  </del> <ins> git clone https://github.com/kubernetes/sample-apiserver /go/src/k8s.io/sample-apiserver &&  cd /go/src/k8s.io/sample-apiserver &&  git checkout tags/v0.17.0 &&  go mod tidy &&  GO111MODULE=on CGO_ENABLED=0 GOARM=${GOARM} GOOS=${OS} GOARCH=${ARCH} go install . &&  </ins> find /go/bin -name sample-apiserver* -exec cp {} ${TARGET}/sample-apiserver ;" # for arm, go install uses /go/bin/linux_arm, so just find the file and copy it to the # root so we can copy it out from this throw away container image from a standard location.

. [1] We've recently switched to Go 1.19, and because of this, the E2E image has started failing to build recently [2] with the error: Using ` directly doesn't work either: [1] [2] The image should be buildable. /sig testing /priority important-soon N/A N/A N/A <details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
please see the PR and lgtm if it works right for you!


Question: "fmt" "testing" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "github.com/stretchr/testify/assert"

What happened: Pods with invalid container-level seccomp profiles allow its init containers to be executed. Pods with unloaded apparmor profiles blocks the entire pod. What you expected to happen: Consistency between both. In one hand AppArmor blocks all pod execution regardless of the containers having or not a profile attached to it. On the other, Seccomp allows for an init container to deploy its own profile - as currently done on . How to reproduce it (as minimally and precisely as possible): Apply the yaml below: Resulting in only the init container from the being executed: ! Anything else we need to know?: AppArmor is currently being target to GA on 1.21 (). Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.0-alpha.0.292+", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-12-12T23:13:21Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node /area security
/cc
/sig security /triage accepted
I think we can consider this as a bug, right? Then we could work on a fix for v1.21 interdependently of the AppArmor GA graduation.
Sounds like a good plan :+1:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/assign


Question: } filteredMinions := v.saltMinionsByRole(minions, "kubernetes-pool") for _, minion := range filteredMinions { <del> fmt.Println("Minion: ", minion.Host, " , ", instance, " IP: ", minion.IP) if minion.Host == instance { </del> <ins> // Due to vagrant not running with a dedicated DNS setup, we return the IP address of a minion as its hostname at this time if minion.IP == instance { </ins> return net.ParseIP(minion.IP), nil } }

In the guestbook example, the redis slaves are instructed to connect to the redis-master using the proxy provided by the minion on which they reside. The minion is designated by its hostname "kubernetes-minion-N". The container cannot resolve this name to an IP address and so slave connections fail. 1) create vagrant cluster 2) create redis-master pod 3) create redis-master service 4) create redis-slave replicationController Verify redis-slave pod locations: list pods ID Image(s) Host Labels Status redis-master-2 dockerfile/redis kubernetes-minion-1/10.245.2.2 name=redis-master Running -3839-11e4-a922- brendanburns/redis-slave kubernetes-minion-1/10.245.2.2 name=redisslave,replicationController=redisSlaveController Running -3839-11e4-a922- brendanburns/redis-slave kubernetes-minion-3/10.245.2.4 name=redisslave,replicationController=redisSlaveController Running Log onto a minion containing a slave (note: vagrant hostnames are minion-N, not kubernetes-minion-N) Find the slave container docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES brendanburns/redis-slave:latest /bin/sh -c About an hour ago Up About an hour k8s--slave.---3839-11e4-a922- docker logs | tail [7] 09 Sep 17:27:08.428  Connecting to MASTER kubernetes-minion-1: [7] 09 Sep 17:27:08.430 # Unable to connect to MASTER: No such file or directory [7] 09 Sep 17:27:09.435  Connecting to MASTER kubernetes-minion-1: sudo nsenter -m -u -n -i -p -t $(docker inspect --format '{{ .State.Pid }}' ) /bin/bash [ root ]$ ping kubernetes-minion-1 ping: unknown host kubernetes-minion-1 [ root ]$ cat /etc/hosts 244.1.4 -3839-11e4-a922- 127.0.0.

1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters [ root ]$ cat nameserver 10.0.2.3 cat /proc/1/environ HOME=/rootPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binHOSTNAME=-3839-11e4-a922-0800279696e1REDISMASTERSERVICEPORT=10000REDISMASTERPORT=tcp://kubernetes-minion-1:10000REDISMASTERPORT6379TCP=tcp://kubernetes-minion-1:10000REDISMASTERPORT6379TCPPROTO=tcpREDISMASTERPORT6379TCPPORT=10000REDISMASTERPORT6379TCPADDR=kubernetes-minion-1SERVICE_HOST=kubernetes-
Tim - are you the right guy to handle this? I think we need to resolve the name of the service host and only give IPs to the minion in env variables. We can't assume the containers have the same DNS set up as the host system.
To be clear -- this example is for the checked in Vagrant stuff? If that is the case would be the guy? I'm trying to catch up on IRC discussions from earlier today.
I can look at this, but there are two possible solutions. The first solution is in the vagrant cluster (or any other local vm setup) we could run a dns name server on the kubernetes-master that can resolve each minion by host name. We would then pass a -dns option to the docker daemon to specify that name server to use in each container. The second solution is that we pass IP address instead of host name and make no assumption or override on the containers specified name server. Is there any obvious negative to this solution? Has much thought been done on the expected Nameserver used by containers spawned by our service? - if you can weigh in, I would be happy to resolve using either approach. My preference is solution two.
So, the vagrant case is a little special as there is no sane DNS set up by default. Another suggestion (3) would be to revert the stuff and just devolve to using ip addresses directly. I would do (3) before (2) before (1).

But I'm cool with any of them. As for the nameserver of the containers that are spawned -- docker currently just takes the of the host and copies that into the container. It doesn't do that for . that is the core of the issue here. If/when we do offer some level of split horizon DNS (I'd love to) we can start exposing an extra layer of DNS on top of whatever is configured for the host. That is part of the "ip per service portal" stuff that is being talked about a lot right now.
If I do (3) there is a piece of code in pod storage that mangles my hostname where I return 10.245.x.x and it asks the cloud provider to resolve the IP address for "10". Since other providers appear to also be returning IP as host, I will look to change this so kubernetes makes no modifications to hostnames that it's given. Look for PR tomorrow.
you're right I'd missed that point. This really only applies to the Vagrant cluster and so a fix to the vagrant setup is sufficient. I also agree with your priorities. Injecting a line into /etc/hosts (if that's reasonable) is the simplest solution. Either reverting or allowing a switch to pass the minion public IP rather than hostname is next. Adding a DNS service to the vagrant master and forcing DNS resolution to the master host is most complicated and least preferable. Thanks both of you for helping me think this through better. Mark
I don't grok what the difference between 2 and 3 is? I sent a PR to docker a while back () to insert arbitrary lines to /etc/hosts. I still think it is a reasonable thing to want to do.
- difference between (2) and (3) is that in the past, the vagrant cloud provider was returning the IP address as the literal hostname for the minion. I had changed this after noticing that it had some ugly side-effects in the CLI, where a call to list pods would report pod ip addresses as IP / nil.

The issue (which I now believe is a bug that I want to change independent of what we do here) is that the code at this line mangles the hostname returned by the cloudprovider: This makes using the IP address as a hostname (10.245.2.2) to have the ugly side-effect of the code then asking the cloudprovider for the IP address of hostname "10", and then the cloudprovider having no idea how to proceed. Independent of what is done in this issue, I want to submit a PR tomorrow to stop that code from manipulating the reported hostname that was provided by the cloudprovider before asking that same cloudprovider to return the IP address using a potentially different input value. so with that backlog (3) means revert vagrant cloudprovider to use IPAddress as hostname, which means when we pass in what we believe is hostname in this environment, it will just work because it is in-fact the IPAddress. (2) means do not pass in hostname but pass in IPAddress by changing more core k8s code. all that being said, there is an option (4) that I am intrigued by, - after following your PR through its final conclusion, it looks like as of docker 1.2.0 you can change the /etc/hosts file of a running container. this introduces an option where the kubelet injects the /etc/hosts file of minion into the running container so it has the /etc/hosts values of the minion. I would need to play with that support a little more to know pros/cons more. I think longer term, we need a plan on what nameserver our containers are expecting to reference, and if they are inherently different in some way. I have run into a number of times the issue where doing things like running docker in docker causes the nested container to default to the nameserver 8.8.8.8, which happens to be blocked on the Red Hat network ;-) For now to unblock I will probably pursue (3) and fix the hostname bug referenced here at the same time. It does make the CLI uglier when working with this environment, but not a huge deal, at least it will function.
Thanks both for the complete explanation of what's happening and for the suggested fixes.

I think in the long run using hostnames in SERVICEHOST and ensuriing either through IP injection into container /etc/hosts or by providing DNS are the best options. I'm not sure if there might still be cases where resolution of SERVICEHOST would still be best done by presenting the IP address or not.


Question: // Reset sends a reset frame, putting the stream into the fully closed state. func (s *Stream) Reset() error { s.conn.removeStream(s) <ins> return s.resetStream() } func (s *Stream) resetStream() error { </ins> s.finishLock.Lock() if s.finished { s.finishLock.Unlock()

Hit this while :
cc/ Andy can you take a look at this failure?
looking now
Working with on a fix
Proposed fix:


Question: attachLimit         int enableTopology      bool enableNodeExpansion bool <del> cleanupHandle       framework.CleanupActionHandle </del> javascriptHooks     map[string]string tokenRequests       []storagev1.TokenRequest requiresRepublish   *bool

Tracking bug for /sig storage /important-longterm /assign
/sig storage /triage accepted
/priority important-longterm


Question: cadvisorapiv2 "github.com/google/cadvisor/info/v2" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" <del> oteltrace "go.opentelemetry.io/otel/trace" </del> v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types"

Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first.

Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in


Question: readonly master_ssh_supported_providers="gce aws kubemark" readonly node_ssh_supported_providers="gce gke aws" <del> readonly master_logfiles="kube-apiserver kube-scheduler rescheduler kube-controller-manager etcd glbc cluster-autoscaler" </del> <ins> readonly master_logfiles="kube-apiserver kube-scheduler rescheduler kube-controller-manager etcd glbc cluster-autoscaler kube-addon-manager" </ins> readonly node_logfiles="kube-proxy" readonly aws_logfiles="cloud-init-output" readonly gce_logfiles="startupscript"

Addon Manager was broken a few times recently by some indirect-related changes(like ) made in the code base. Addon Manager's logs would be very helpful for narrowing down the problem. And I don't think there is an obvious way to retrieve Addon Manager's logs for Jenkins tests currently. This can be achieved in the same way as what we did for the components like and (mounting /var/log). I will try it through this way for addon manager. cc


Question: {% if pillar.get('network_provider', '').lower() == 'opencontrail' %} {% set network_plugin = "--network-plugin=opencontrail" %} {% elif pillar.get('network_provider', '').lower() == 'cni' %} <del> {% set network_plugin = "--network-plugin=cni --cni-bin-dir=/etc/cni/net.d/" %} {%elif pillar.get('network_policy_provider', '').lower() == 'calico' and grains['roles'][0] != 'kubernetes-master' -%} </del> <ins> {% set network_plugin = "--network-plugin=cni --cni-conf-dir=/etc/cni/net.d/ --cni-bin-dir=/home/kubernetes/bin/" %} {% elif pillar.get('network_policy_provider', '').lower() == 'calico' and grains['roles'][0] != 'kubernetes-master' %} </ins> {% set network_plugin = "--network-plugin=cni --cni-conf-dir=/etc/cni/net.d/ --cni-bin-dir=/home/kubernetes/bin/" %} {% elif pillar.get('network_provider', '').lower() == 'kubenet' %} <del> {% set network_plugin = "--network-plugin=kubenet" -%} </del> <ins> {% set network_plugin = "--network-plugin=kubenet --network-plugin-dir=/home/kubernetes/bin/" -%} </ins> {% endif -%} # Don't pipe the --hairpin-mode flag by default. This allows the kubelet to pick

Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): debian, gci Install tools: cluster/kube- What happened: Creating a cluster with GCI uses the directory for CNI plugins. Creating a cluster with debian uses the directory for CNI plugins, which is read-only on GCI. This makes it impossible to create a DaemonSet which works for both image types. What you expected to happen: The CNI installation directory should be the same across all image types. In this case, it should be . CC
/sig network
Some related work here:
Ok, I approved this one for 1.7 since It helps clean up our CVM based tests. The cleanup code is also self-contained, shouldn't cause any issues on other cloud providers. But will double check through the pr.


Question: if err != nil { t.Fatalf("Failed to create replication controller: %v", err) } <del> pod := newMatchingPod("pod1", ns.Name) </del> <ins> podName := fmt.Sprintf("pod%d", i) pod := newMatchingPod(podName, ns.Name) </ins> pod.OwnerReferences = tc.existingOwnerReferences(rc) _, err = podClient.Create(pod) if err != nil {

https://k8s- Failed: TestAdoption {integration}
https://k8s- Failed: TestAdoption {integration}
https://k8s- Failed: TestAdoption {integration}
https://k8s- Failed: TestAdoption {integration}
The problem is that the rc manager hasn't observed pod1, so it creates another pod and scales down, pod1 might get deleted. I'm working on the fix.


Question: }) }) }) <ins> ginkgo.Context("Kubelet", func() { ginkgo.It("should correctly account for terminated pods after restart", func() { node := getLocalNode(f) cpus := node.Status.Allocatable[v1.ResourceCPU] numCpus := int((&cpus).Value()) if numCpus < 1 { e2eskipper.Skipf("insufficient CPU available for kubelet restart test") } // create as many restartNever pods as there are allocatable CPU // nodes; if they are not correctly accounted for as terminated // later, this will fill up all node capacity podCountRestartNever := numCpus ginkgo.By(fmt.Sprintf("creating %d RestartNever pods on node", podCountRestartNever)) restartNeverPods := newTestPods(podCountRestartNever, false, imageutils.GetE2EImage(imageutils.BusyBox), "restart-kubelet-test") for _, pod := range restartNeverPods { pod.Spec.RestartPolicy = "Never" pod.Spec.Containers[0].Command = []string{"echo", "hi"} pod.Spec.Containers[0].Resources.Limits = v1.ResourceList{ v1.ResourceCPU: resource.MustParse("1"), } } createBatchPodWithRateControl(f, restartNeverPods, podCreationInterval) defer deletePodsSync(f, restartNeverPods) completedPods := waitForPods(f, podCountRestartNever, time.Minute) if len(completedPods) < podCountRestartNever { framework.Failf("Failed to run sufficient restartNever pods, got %d but expected %d", len(completedPods), podCountRestartNever) } podCountRestartAlways := (numCpus / 2) + 1 ginkgo.By(fmt.Sprintf("creating %d RestartAlways pods on node", podCountRestartAlways)) restartAlwaysPods := newTestPods(podCountRestartAlways, false, imageutils.GetPauseImageName(), "restart-kubelet-test") for _, pod := range restartAlwaysPods { pod.Spec.Containers[0].Resources.Limits = v1.ResourceList{ v1.ResourceCPU: resource.MustParse("1"), } } createBatchPodWithRateControl(f, restartAlwaysPods, podCreationInterval) defer deletePodsSync(f, restartAlwaysPods) numAllPods := podCountRestartNever + podCountRestartAlways allPods := waitForPods(f, numAllPods, startTimeout) if len(allPods) < numAllPods { framework.Failf("Failed to run sufficient restartAlways pods, got %d but expected %d", len(allPods), numAllPods) } ginkgo.By("killing and restarting kubelet") // We want to kill the kubelet rather than a graceful restart startKubelet := stopKubelet() startKubelet() // If this test works correctly, each of these pods will exit // with no issue. But if accounting breaks, pods scheduled after // restart may think these old pods are consuming CPU and we // will get an OutOfCpu error. ginkgo.By("verifying restartNever pods succeed and restartAlways pods stay running") for start := time.Now(); time.Since(start) < startTimeout; time.Sleep(10 * time.Second) { postRestartRunningPods := waitForPods(f, numAllPods, time.Minute) if len(postRestartRunningPods) < numAllPods { framework.Failf("less pods are running after node restart, got %d but expected %d", len(postRestartRunningPods), numAllPods) } } }) }) </ins> })

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Restart: Always pods went into OutOfCpu error state after kubelet restart. Pods should remain running as node had sufficient capacity for them. Completed, RestartNever pod should not revert from Completed to OutOfCpu. Launch kubelet with: Run the following script against the local cluster: <details</detailsObserve pod status: Restart the kubelet: Observe pods: xref /sig node Kubernetes version (use ): v1.23.0-alpha.3.183+ - Cloud provider or hardware configuration: Local machine OS (e.g: ) & Kernel (e.g. ):Install tools: local-up- Network plugin and version (if this is a network-related bug): N/A Others:
/triage accepted /priority critical-urgent Reproducible on 1.22
From the reproducer: yields However, in the kubelet logs, it sees 7100m cpu allocated?
Looks like the topology admit handler thinks that the completed pods are using CPU which is why when it gets to the 8th it fails it with OutOfCpu. Must be a bug in there somewhere.
/assign
Unclear if could be related but linking for posterity
Confirmed this only affects 1.22, repro on 1.21 has expected behaviour: /sig scheduling I'll start a bisect.
is
Added some logging With this we see I think that the terminated pods are getting filtered, but for whatever reason they are still getting their resource requests to NodeInfo?
points out that my logging language was a bit backwards and that the completed pods should not have been returned, because they already successfully completed and exited. This code is suspect: The logs show that the completed pods are never filtered, so they are failing one of these checks. I am going to try to write a test case that hits this issue so we can test fixes against it.

The completed pods definitely show up as complete, but for some reason they do not get filtered (note I have changed my patch above for the log lines to make sense now): So I guess isPodTerminationRequested=true on already exited pods, but I would only expect that to be true if the pod was actively in the process of termination and not already exited. cc Log patch: <details</details
I tried changing to but apparently it thinks that's true :( Edit: doesn't work either.
The bug is that the only component in the kubelet that is allowed to authoritatively state a container is terminated (pod worker) takes non-zero time to do so during startup (because we have to call kill container to guarantee it's stopped). This races with the pod config loop and admission, which sends all the containers at once. In order for the race to be eliminated, some equivalent of a sync has to be completed during startup on terminal but potentially non-terminated pod workers BEFORE other adds are processed. That could be running HandlePodCleanups once all sources are synced but before HandlePod* methods are called, but HandlePodAdditions() cannot be invoked until after the pod worker has had a chance to verify all terminal pods are actually terminal. EDIT: Another potential approach is for the pod worker to hold starting pods until SyncKnownPods observes all terminal pods are accounted for at least once, then allowing starts.
Also, we appear to be missing a kubelet restart test that verifies admission with completed pods (which I thought was on our list of "missing e2e tests that would have caught these regressions" already). Such a test would have to: pods that consume resources for admission until they are running some of them with failed and` successful states the kubelet some of the running pods the kubelet none of the terminal pods are restarted, all running pods are restarted, and that only non-terminal pods are accounted to admission


Question: - endpoint: replaceAutoscalingV2NamespacedHorizontalPodAutoscalerStatus reason: optional feature link: https://github.com/kubernetes/kubernetes/pull/106595#issuecomment-1006015756 <ins> - endpoint: logFileHandler reason: optional feature link: https://github.com/kubernetes/kubernetes/issues/108675 - endpoint: logFileListHandler reason: optional feature link: https://github.com/kubernetes/kubernetes/issues/108675 </ins>

We have a question regarding the eligibility of the following 2 endpoints for conformance; logFileListHandler logFileHandler We found this PR Don’t test the debug /logs endpoint on GKE. that disables testing of /log endpoint for GKE. Is this still valid or should the exclusion of GKE be reconsidered? Does this provider requirement make these 2 log endpoints ineligible for conformance?
/triage accepted /sig architecture /area conformance
confirmed that these endpoints should no be eligible for conformance. They should be considered an operational tool, not related to application consistency across distros.


Question: } // dsFromManifest reads a .json/yaml file and returns the daemonset in it. <del> func dsFromManifest(fileName string) *extensions.DaemonSet { </del> <ins> func dsFromManifest(url string) *extensions.DaemonSet { </ins> var controller extensions.DaemonSet <del> framework.Logf("Parsing ds from %v", fileName) data := generated.ReadOrDie(fileName) </del> <ins> framework.Logf("Parsing ds from %v", url) var response *http.Response var err error for i := 1; i <= 5; i++ { response, err = http.Get(url) if err == nil && response.StatusCode == 200 { break } time.Sleep(time.Duration(i) * time.Second) } Expect(err).NotTo(HaveOccurred()) Expect(response.StatusCode).To(Equal(200)) defer response.Body.Close() data, err := ioutil.ReadAll(response.Body) Expect(err).NotTo(HaveOccurred()) </ins> json, err := utilyaml.ToJSON(data) Expect(err).NotTo(HaveOccurred())

xref: /sig node /area test
I am working on this.
Can you update this with the types of tests that you are adding? Is this similar to ?
No, this is for running the existing GPU e2e tests on GKE which are currently running on only GCE.
The referenced PRs appear to have merged, closing.


Question: // create new iface out, err = b.exec.Run("iscsiadm", "-m", "iface", "-I", newIface, "-o", "new") if err != nil { <del> lastErr = fmt.Errorf("iscsi: failed to create new iface: %s (%v)", string(out), err) return lastErr </del> <ins> exit, ok := err.(utilexec.ExitError) if ok && exit.ExitStatus() == iscsiadmErrorSessExists { klog.Infof("iscsi: there is a session already logged in with iface %s", newIface) } else { lastErr = fmt.Errorf("iscsi: failed to create new iface: %s (%v)", string(out), err) return lastErr } </ins> } // update new iface records for key, val := range params {

What happened: In order to communicate with the target, the iscsi plugin creates a new iface. If for some reason an iface with the same name already exists in the node, this command will re-create it and no error will be reported. However, if the iface already exists and is being used by a session, this command will fail and report the error code 15 (). That's because it can't re-create the iface while it's in use by a session. What you expected to happen: The plugin could go ahead and use the existing iface instead of failing. How to reproduce it (as minimally and precisely as possible): pod1 pod2 Here, will re-use the session/iface from the step above. pod1 At this point, k8s will try to delete the session/iface used by pv1. However, it'll fail because the session is currently in use by pv2. pod1 Here the pod's status will be stuck at : Templates used: Anything else we need to know?: problem happens when a custom initiator is used, and it was reported to me that it's necessary in some set-ups in order to "restrict access to LUNs of the storage". to reproduce manually (assuming there's a session using the iface): /sig storage /CC
/assign
is this related to cc
If iface is reused, wouldn't it try to delete it once last volume (from kubelet perspective) is detached?
No


Question: } // Returns the first security group for an instance, or nil <del> // We only create instances with one security group, so we warn if there are multiple or none func findSecurityGroupForInstance(instance *ec2.Instance) *string { var securityGroupId *string for _, securityGroup := range instance.SecurityGroups { if securityGroup == nil || securityGroup.GroupId == nil { // Not expected, but avoid panic glog.Warning("Unexpected empty security group for instance: ", orEmpty(instance.InstanceId)) </del> <ins> // We only create instances with one security group, so we don't expect multiple security groups. // However, if there are multiple security groups, we will choose the one tagged with our cluster filter. // Otherwise we will return an error. func findSecurityGroupForInstance(instance *ec2.Instance, taggedSecurityGroups map[string]*ec2.SecurityGroup) (*ec2.GroupIdentifier, error) { instanceID := aws.StringValue(instance.InstanceId) var best *ec2.GroupIdentifier for _, group := range instance.SecurityGroups { groupID := aws.StringValue(group.GroupId) if groupID == "" { glog.Warningf("Ignoring security group without id for instance %q: %v", instanceID, group) </ins> continue } <del> if securityGroupId != nil { // We create instances with one SG glog.Warningf("Multiple security groups found for instance (%s); will use first group (%s)", orEmpty(instance.InstanceId), *securityGroupId) </del> <ins> if best == nil { best = group </ins> continue <ins> } _, bestIsTagged := taggedSecurityGroups[*best.GroupId] _, groupIsTagged := taggedSecurityGroups[groupID] if bestIsTagged && !groupIsTagged { // best is still best } else if groupIsTagged && !bestIsTagged { best = group </ins> } else { <del> securityGroupId = securityGroup.GroupId </del> <ins> // We create instances with one SG // If users create multiple SGs, they must tag one of them as being k8s owned return nil, fmt.Errorf("Multiple security groups found for instance (%s); ensure the k8s security group is tagged", instanceID) </ins> } } <del> if securityGroupId == nil { glog.Warning("No security group found for instance ", orEmpty(instance.InstanceId)) </del> <ins> if best == nil { glog.Warning("No security group found for instance ", instanceID) </ins> } <del> return securityGroupId </del> <ins> return best, nil } // Return all the security groups that are tagged as being part of our cluster func (s *AWSCloud) getTaggedSecurityGroups() (map[string]*ec2.SecurityGroup, error) { request := &ec2.DescribeSecurityGroupsInput{} filters := []*ec2.Filter{} request.Filters = s.addFilters(filters) groups, err := s.ec2.DescribeSecurityGroups(request) if err != nil { return nil, fmt.Errorf("error querying security groups: %v", err) } m := make(map[string]*ec2.SecurityGroup) for _, group := range groups { id := aws.StringValue(group.GroupId) if id == "" { glog.Warningf("Ignoring group without id: %v", group) continue } m[id] = group } return m, nil </ins> } // Open security group ingress rules on the instances so that the load balancer can talk to them

If there are two security groups on an instance, we currently make an arbitrary choice. We should choose based on tags, as we normally do. For back-compat, we need to cope if there is one security group and it is not tagged. We should make sure kube-up tags our security groups, though it isn't really required for kube-up, but it is just good behaviour. cf


Question: klog.Errorf("iscsi: failed to mkdir %s, error", globalPDPath) return err } <ins> if b.volumeMode == v1.PersistentVolumeFilesystem { notMnt, err := b.mounter.IsLikelyNotMountPoint(globalPDPath) if err != nil { return err } if !notMnt { // The volume is already mounted, therefore the previous WaitForAttach must have // persisted the volume metadata. In addition, the metadata is actually *inside* // globalPDPath and we can't write it here, because it was shadowed by the volume // mount. klog.V(4).Infof("Skipping persistISCSI, the volume is already mounted at %s", globalPDPath) return nil } } </ins> // Persist iscsi disk config to json file for DetachDisk path return util.persistISCSIFile(*(b.iscsiDisk), globalPDPath) }

What happened: Running two pods that use the same iSCSI volume, one of them fails with: What you expected to happen: Both pods should be running Anything else we need to know?: It's caused by iscsi volume plugin persiting iSCSI settings into the mount directory itself (e.g. ): volume plugin creates the global mount directory. volume plugin creates there. volume plugin mounts the volume there. This shadows the file, but it's fine, as long as no other pod uses the same volume on the same node. creates a second pod that uses the volume. volume plugin tries to create in the global volume directory. Since the volume is mounted, it will either writes the file to the volume (!) or fails, if the volume is read-only.
/sig storage /assign /priority impotant-soon
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig storage /assign /priority impotant-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
Our e2e did not catch this, because they always create only one pod when testing read-only volumes.


Question: netConf, cniNet := network.NetworkConfig, network.CNIConfig glog.V(4).Infof("About to del CNI network %v (type=%v)", netConf.Name, netConf.Plugins[0].Network.Type) err = cniNet.DelNetworkList(netConf, rt) <del> if err != nil { </del> <ins> // The pod may not get deleted successfully at the first time. // Ignore "no such file or directory" error in case the network has already been deleted in previous attempts. if err != nil && !strings.Contains(err.Error(), "no such file or directory") { </ins> glog.Errorf("Error deleting network: %v", err) return err }

/kind bug What happened: During pod teardown, we sometimes see the following error-level kubelet logs: What you expected to happen: Clean pod shutdown, no error-level logs. How to reproduce it (as minimally and precisely as possible): We've got two kube crons that run every minute, across 12 regions in production -- so related pods daily -- and we see this about 3-12 times per day. Anything else we need to know?: Relevant kubelet log context of a cron pod that manifested this error: Controller manager for the pod: Scheduler for the pod: Cron yaml: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): 16.04.1 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-89-generic -Ubuntu Install tools: N/A Others: N/A
/sig network
Same problem here. Every time I try to of my deploy, the old pod cannot be terminated successfully. below is output of
Have you checked the ns net files of those failed-to-deleted pods? /cc Could we just ignore these errors?
Sorry, yeah, that seems like a fair point to note -- the directories are in fact being deleted. Seems like it's either functionally succeeding yet generating error-level logs, or perhaps generating error-level logs and then succeeding on internal retry.
I prefer the later one. Perhaps kubelet does not kill the pod completely at the first time. Such error may occur during a retry.
So this is just an overly scary log message?
Yes, I think so. In the first attempt to delete the pod, the ns network file has already been deleted. But the pod does not get deleted successfully. Then "no such file or directory" will be raised in the second attempt.


Question: NonMergingIntList []int MergeItemPtr      *MergeItem `patchStrategy:"merge" patchMergeKey:"name"` SimpleMap         map[string]string <ins> ReplacingItem     runtime.RawExtension `patchStrategy:"replace"` </ins> } var mergeItem MergeItem

Extracted from The RawExtension type allows an API object to inline arbitrary JSON. strategicpatch.CreateTwoWayMergePatch returns an error trying to map that inlined JSON to struct fields. It should be able to compute a patch for an object containing a field of that type.
cc
We need to decide what strategic merge patch should do in the following scenarios: unknown fields (can be encountered when the API or a user-defined object has a new field the compiled-in struct doesn't have yet) unmergeable fields (like RawExtension) cc
The current behavior is: unknown fields: trying to merge a unknown field will fail when lookup the metadata in the struct tags unmergeable fields: will be replaced as raw byte array Maybe we can use generic json merge for unmergeable fields?
Did that change since 1.5? In 1.5, we were getting an error during patch computation for objects with a RawExtension field
Sorry. This behavior is in 1.4.7. e.g . It will error in 1.5.
Yeah, the field gets replaced. Even in 1.4.7, an unknown field (like initcontainers in pods) or an object including a RawExtension (anything with a field of type or ) would error
My personal favor would be to instruct the patch to ignore such field, seems the cleanest approach to me. Similarly to how we allow annotating deletion and merging.
wouldn't it be confusing to submit a patch with stuff in that field, have the patch accepted, and have none of your submitted changes take effect? seems like the least surprising thing would be to treat struct fields like or as
How about both, iow. having as a default and as an opt-in. This would allow the resource author to decide how his resource should be treated. Eventually on we could return the ignored fields. Having said that we could start with right away. I think I could work on a patch implementing that if majority agrees (
I think is a good place to start when computing/applying patches for known struct fields like RawExtension. I'll open a separate issue for client-side handling of computing patches for unknown fields
Ignore the field sounds like a really good solution. WDYT of printing a warning to stdeer?
what does ignoring instead of replacing get us?
Sorry I was reading the comment on my phone between other things. Ignore my comment.


Question: RUN CGO_ENABLED=0 GOOS=linux GO111MODULE=on go build -a -ldflags "-w" -o /elasticsearch_logging_discovery /elasticsearch_logging_discovery.go <del> FROM docker.elastic.co/elasticsearch/elasticsearch-oss:7.1.1 </del> <ins> FROM docker.elastic.co/elasticsearch/elasticsearch-oss:7.2.0 </ins> VOLUME ["/data"] EXPOSE 9200 9300

What happened: On latest fluentd-elasticsearch, the elasticsearch replicas seems failed to form a valid cluster with the service discovery helper written in golang. What you expected to happen: Successfully boot up an elasticsearch cluster I guess. How to reproduce it (as minimally and precisely as possible): Just use the yamls in and the elasticsearch-logging pods will keep spamming warning messages. I think thats because options are deprecated in elasticsearch 7.x.x Anything else we need to know?: IMHO I think thats all. Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig instrumentation
Thanks


Question: MASTER_SIZE=n1-standard-1 MINION_SIZE=n1-standard-1 NUM_MINIONS=${NUM_MINIONS:-4} <ins> MINION_DISK_TYPE=pd-standard MINION_DISK_SIZE=10GB </ins> # TODO(dchen1107): Filed an internal issue to create an alias # for containervm image, so that gcloud will expand this # to the latest supported image.

The node disk type should be an option in the cluster/gce/config-*.sh files. The nodes get created here:
And also the disk size.
How to do this for aws ? I am stuck at 10GB minion
It . A pull request adding it would be very welcome!
please check this thread
Hello there, can I do something to make this available in the command ? I'd like to us an SSD drive :)


Question: } } } <ins> const listTypeResourceSchema = ` apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: foos.test spec: group: test names: kind: Foo listKind: FooList plural: foos singular: foo scope: Cluster versions: - name: v1 schema: openAPIV3Schema: type: object properties: numArray: type: array x-kubernetes-list-type: set items: type: object served: true storage: true - name: v2 schema: openAPIV3Schema: type: object properties: numArray2: type: array ` func TestStatusStrategyValidateUpdate(t *testing.T) { crdV1 := &apiextensionsv1.CustomResourceDefinition{} err := yaml.Unmarshal([]byte(listTypeResourceSchema), &crdV1) if err != nil { t.Fatalf("unexpected decoding error: %v", err) } t.Logf("crd v1 details: %v", crdV1) crd := &apiextensions.CustomResourceDefinition{} if err = apiextensionsv1.Convert_v1_CustomResourceDefinition_To_apiextensions_CustomResourceDefinition(crdV1, crd, nil); err != nil { t.Fatalf("unexpected convert error: %v", err) } t.Logf("crd details: %v", crd) strategy := statusStrategy{} kind := schema.GroupVersionKind{ Version: crd.Spec.Versions[0].Name, Kind:    crd.Spec.Names.Kind, Group:   crd.Spec.Group, } strategy.customResourceStrategy.validator.kind = kind ss, _ := structuralschema.NewStructural(crd.Spec.Versions[0].Schema.OpenAPIV3Schema) strategy.structuralSchemas = map[string]*structuralschema.Structural{ crd.Spec.Versions[0].Name: ss, } ctx := context.TODO() tcs := []struct { name    string old     *unstructured.Unstructured obj     *unstructured.Unstructured isValid bool }{ { name:    "bothValid", old:     &unstructured.Unstructured{Object: map[string]interface{}{"apiVersion": "test/v1", "kind": "Foo", "numArray": []interface{}{1, 2}}}, obj:     &unstructured.Unstructured{Object: map[string]interface{}{"apiVersion": "test/v1", "kind": "Foo", "numArray": []interface{}{1, 3}, "metadata": map[string]interface{}{"resourceVersion": "1"}}}, isValid: true, }, { name:    "change to invalid", old:     &unstructured.Unstructured{Object: map[string]interface{}{"apiVersion": "test/v1", "kind": "Foo", "spec": "old", "numArray": []interface{}{1, 2}}}, obj:     &unstructured.Unstructured{Object: map[string]interface{}{"apiVersion": "test/v1", "kind": "Foo", "spec": "new", "numArray": []interface{}{1, 1}, "metadata": map[string]interface{}{"resourceVersion": "1"}}}, isValid: false, }, { name:    "change to valid", old:     &unstructured.Unstructured{Object: map[string]interface{}{"apiVersion": "test/v1", "kind": "Foo", "spec": "new", "numArray": []interface{}{1, 1}}}, obj:     &unstructured.Unstructured{Object: map[string]interface{}{"apiVersion": "test/v1", "kind": "Foo", "spec": "old", "numArray": []interface{}{1, 2}, "metadata": map[string]interface{}{"resourceVersion": "1"}}}, isValid: true, }, { name:    "keeps invalid", old:     &unstructured.Unstructured{Object: map[string]interface{}{"apiVersion": "test/v1", "kind": "Foo", "spec": "new", "numArray": []interface{}{1, 1}}}, obj:     &unstructured.Unstructured{Object: map[string]interface{}{"apiVersion": "test/v1", "kind": "Foo", "spec": "old", "numArray": []interface{}{1, 1}, "metadata": map[string]interface{}{"resourceVersion": "1"}}}, isValid: true, }, } for _, tc := range tcs { errs := strategy.ValidateUpdate(ctx, tc.obj, tc.old) if tc.isValid && len(errs) > 0 { t.Errorf("%v: unexpected error: %v", tc.name, errs) } if !tc.isValid && len(errs) == 0 { t.Errorf("%v: unexpected non-error", tc.name) } } } </ins>

doesn't do the same embedded resource and x-kubernetes-list-type "map" and "set" invariant validation that happens in custom resource create/update validation. Does that mean embedded objects and list-type fields under custom resource status can contain invalid data? if so, that needs to be fixed in a way that tolerates existing invalid data (ratcheting validation) Originally posted by in /sig api-machinery /kind bug /priority important-longterm
/triage accepted
I would like to collaborate for fixing the validation issue for custom resources and tolerate the existing invalid data. Could you please guide me with some code pointers or files and elaborate on how can I proceed for fixing this bug.
I haven't tested this, but something along the lines of this: we'd need tests to exercise the following scenarios on a CRD that specifies a status subresource with a list property with x-kubernetes-list-type=map and x-kubernetes-list-type=set a custom resource instance that already has duplicate items in the status property list can be updated successfully while still having duplicate items in the list (validation doesn't block an invalid → invalid status update) a custom resource instance that already has duplicate items in the status property list can be updated successfully to set unique items in the list (validation doesn't block an invalid → valid status update) a custom resource instance that does not have duplicate items in the status property list cannot be updated to set duplicate items in the list (validation blocks a valid → invalid status update)
since it's been a couple of months, i'm going to try and take a stab at this /assign
/assign
Hi k8s 1.25 Bug Triage Shadow here, just checking in to see if this is still on track for k8s 1.25? Thank you.
has stalled needing tests and a rebase
Thanks for the update.
Hello with the code freeze approaching, is this still on track for v1.25? Thanks!
not really, the status on is the same (needs tests)
/milestone clear


Question: func (f *fakeVMSet) GetDataDisks(nodeName types.NodeName) ([]compute.DataDisk, error) { return nil, fmt.Errorf("unimplemented") } <ins> func (f *fakeVMSet) GetProvisioningStateByNodeName(name string) (string, error) { return "", fmt.Errorf("unimplemented") } </ins>

Implement proposal for Azure We should implement interface in Azure cloud provider. returns true if the instance is in stopped state to detach volumes. But if instance is deleted we should not return error. Related features: Related proposal: Related PRs: /kind feature /sig azure /cc
- can you assign me on this one ? I already have a PR in flight.
That's great! /assign just loop in for this RP review, thanks.
GitHub didn't allow me to assign the following users: yastij. Note that only and repo collaborators can be assigned. <detailsIn response to : That's great! /assign just loop in for this RP review, thanks. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
o, you are not a k8s member yet, anyway, you could send out the PR for this feature, related to this issue, thanks.
Thanks, looking forward to your PR.


Question: return true } <ins> switch result { case results.Success: ProberResults.With(w.proberResultsSuccessfulMetricLabels).Inc() case results.Failure: ProberResults.With(w.proberResultsFailedMetricLabels).Inc() default: ProberResults.With(w.proberResultsUnknownMetricLabels).Inc() } </ins> if w.lastResult == result { w.resultRun++ } else {

Currently the kubelet exports liveness/readiness probe metric information at an . The metric data looks like this: This is incredibly useful metric information, however, it would be better if these metrics were counters rather than gauges. Currently a success probe is registered as a '0' value and a failure is registered as a '1' value. However, depending on your prometheus scrape interval, you can lose bits of information. It would be better if we emitted these metrics as counters. /sig instrumentation node
Perhaps a new counters metrics to record the the number of failure is a better option. Because the gauge metrics is easy to watch and alert. I can do it if agree with it. /assign
This is a good point. However, if we really did want to use it for this purpose, it would be better if the gauge metric reported '0' for success or else the current running count of failures for a given prober endpoint. With the current implementation, we are needlessly dropping some bits. It may be useful to have both. One can provide a more real time signal for alerting but the other gives you better historical data on master component probe results.
do you have a preference on gauge vs counter here?
For our purposes a counter makes much more sense, because we want to use it to figure out the proportion of time something was healthy (e.g. for calculating availability SLOs). It also means we can have a much higher fidelity measurement - e.g. we could ramp up the healthchecks to every 5s but only scrape the results every minute. Can you point at some examples of people doing this, so I can better understand how these metrics are being used? It seems like using point-in-time samples of health is going to result in noisy alerts, unless you do more work elsewhere to aggregate ("alert if this metric was 0 for X minutes" - which then runs the risk of missing samples, as mentioned)
if the metrics value is 0 or 1, we can simple set prometheus rule as just like you mentioned.
Yes, I agree that the alert rule would be simple but it also feels prone to be noisy (as mentioned). A failed probe is not necessarily alert-worthy, since the kubelet defers action until the ; by default, the kubelet doesn't do anything until we have 3 consecutive failures.

Yep, so with in alert rule, it will not fire alert noisily. /cc
All the facts have already been stated, I'm in favor of removing the gauges and add counters (I don't see a state that we can't interpolate from the counter that the gauge displays today, yet the counter is more reliable in alerting aspects as mentioned).
sure. As a counter, how to represent successful probe? If the probe is successful, no action is taken, otherwise plus 1 to this metrics ?
I'm thinking something along the lines of:
Yeah, I was thinking along the same lines (i.e. using the result of the probe as a label).
Also now that we're talking about this and inspired by the metrics overhaul of 1.14, we should probably standardize these label values across Kubernetes. I'll add it to the next agenda.
A little off topic but wondering if someone could help guide me on how to access this endpoint? I've tried both k8s 1.13 and 1.14 and cannot access metrics/probes (i can only hit /metrics) via the api server http endpoint. Is it exposed somewhere else, or is it a flag I have to enable?
I had this issue for few days. What you should do is in your config you have to add "metricspath: /metrics/cadvisor". This will scrape out all container metrics that you would like to see. global: scrapeinterval: 5s evaluationinterval: 5s rulefiles: /etc/prometheus-rules/cpu- /etc/prometheus-rules/instance- scrapeconfigs: jobname: 'kubernetes-nodes' tlsconfig: cafile: bearertokenfile: kubernetessdconfigs: role: node metricspath: /metrics/cadvisor relabelconfigs: sourcelabels: [address] regex: '(.*):' replacement: '${1}:' targetlabel: address
This isn't an apiserver metric, it's actually a kubelet metric, so it's exposed on a different port. Are you using 8080? I think it should be or something like that.
@ munsayac13 thanks for the suggestion.

This is what I ended up doing to solve this: update the kubelet servicemonitor (e.g kubectl edit servicemonitor kubelet) this: bearerTokenFile: honorLabels: true interval: 30s path: /metrics/probes port: https-metrics scheme: https tlsConfig: insecureSkipVerify: true
Hi folks, I wonder if we could add the old gauge metrics back for backward compatibility sake. We just realized this change breaks some of our components that relies on the old metrics during the migration to v1.15.x.
We don’t have stability guarantees on these metrics (or any for that matter). I don’t think we’ll be reintroducing these, but we’re introducing a metrics stability framework for these things to not happen again or have the same deprecation process as kubernetes APIs in the future. Sorry for the inconvenience!
that is understandable.


Question: tc.runTest(t) } <ins> func TestReplicaCalcMissingMetricsScaleDownTargetOver100(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  4, expectedReplicas: 2, podReadiness:     []v1.ConditionStatus{v1.ConditionFalse, v1.ConditionTrue, v1.ConditionTrue, v1.ConditionTrue}, resource: &resourceInfo{ name:     v1.ResourceCPU, requests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("2.0"), resource.MustParse("2.0")}, levels:   makePodMetricLevels(200, 100, 100), targetUtilization:   300, expectedUtilization: 6, expectedValue:       numContainersPerPod * 100, }, } tc.runTest(t) } </ins> func TestReplicaCalcDuringRollingUpdateWithMaxSurge(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  2,

Change the fallback consumption from to when pod metrics are missing on a scale down. Background from : The fallback value of 100% is supposed to be a safe overestimate of the desired value to dampen the magnitude of a potential scale down. However, if a metric target is set to more than 100% then this fallback typically becomes an underestimate that amplifies a potential scale down. Note that a metric target of over 100% is a valid HPA configuration. We could choose a fallback of instead. However, any safety margin would be arbitrary and I think it's better to opt for simplicity vs. precision in this corner case.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
/assign
/assign


Question: genericfeatures.APIServerTracing: {Default: true, PreRelease: featuregate.Beta}, <ins> genericfeatures.APIServingWithRoutine: {Default: true, PreRelease: featuregate.Beta}, </ins> genericfeatures.ConsistentListFromCache: {Default: false, PreRelease: featuregate.Alpha}, genericfeatures.CustomResourceValidationExpressions: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.31

Optimize the memory usage of goroutine stack for serving watch requests. According to the of handling a watch request, the stack memory is 0x3768, which is very close to 16 KiB and Golang runtime will double it to 32 KiB once exceeded. In fact, it does exceed if we print the traceback deeper and take the fact that extra stack memory reserved by Golang itself into consideration. A naive optimization is to run as much as preparation handlers in a separate goroutine and then move the actual serving to another goroutine, so the former routine can free the stack memory earlier and make it more memory efficient. Given the extensive use of watch requests, 32 KiB per watch could add up to a very high amount of memory overhead for . A typical watch request commonly lasts for 30s or longer, which means this memory will be active for quite long. I see a great improvement in memory efficiency to do this optimization.
/cc
/sig api-machinery
/cc if you would like to take a look when you have time :) Thank you /triage accepted


Question: sinkWatcher.Stop() logWatcher.Stop() } <ins> func TestEventfNoNamespace(t *testing.T) { testPod := &api.Pod{ ObjectMeta: api.ObjectMeta{ SelfLink: "/api/version/pods/foo", Name:     "foo", UID:      "bar", }, } testRef, err := api.GetPartialReference(testPod, "desiredState.manifest.containers[2]") if err != nil { t.Fatal(err) } table := []struct { obj          runtime.Object reason       string messageFmt   string elements     []interface{} expect       *api.Event expectLog    string expectUpdate bool }{ { obj:        testRef, reason:     "Started", messageFmt: "some verbose message: %v", elements:   []interface{}{1}, expect: &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "foo", Namespace: "default", }, InvolvedObject: api.ObjectReference{ Kind:       "Pod", Name:       "foo", Namespace:  "", UID:        "bar", APIVersion: "version", FieldPath:  "desiredState.manifest.containers[2]", }, Reason:  "Started", Message: "some verbose message: 1", Source:  api.EventSource{Component: "eventTest"}, Count:   1, }, expectLog:    `Event(api.ObjectReference{Kind:"Pod", Namespace:"", Name:"foo", UID:"bar", APIVersion:"version", ResourceVersion:"", FieldPath:"desiredState.manifest.containers[2]"}): reason: 'Started' some verbose message: 1`, expectUpdate: false, }, } for _, item := range table { called := make(chan struct{}) testEvents := testEventSink{ OnCreate: func(event *api.Event) (*api.Event, error) { returnEvent, _ := validateEvent(event, item.expect, t) if item.expectUpdate { t.Errorf("Expected event update(), got event create()") } called <- struct{}{} return returnEvent, nil }, OnUpdate: func(event *api.Event) (*api.Event, error) { returnEvent, _ := validateEvent(event, item.expect, t) if !item.expectUpdate { t.Errorf("Expected event create(), got event update()") } called <- struct{}{} return returnEvent, nil }, } eventBroadcaster := NewBroadcaster() sinkWatcher := eventBroadcaster.StartRecordingToSink(&testEvents) logWatcher1 := eventBroadcaster.StartLogging(t.Logf) // Prove that it is useful logWatcher2 := eventBroadcaster.StartLogging(func(formatter string, args ...interface{}) { if e, a := item.expectLog, fmt.Sprintf(formatter, args...); e != a { t.Errorf("Expected '%v', got '%v'", e, a) } called <- struct{}{} }) recorder := eventBroadcaster.NewRecorder(api.EventSource{Component: "eventTest"}) recorder.Eventf(item.obj, item.reason, item.messageFmt, item.elements...) <-called <-called sinkWatcher.Stop() logWatcher1.Stop() logWatcher2.Stop() } } </ins>

Any node object related events from the kubelet does not get recorded in the apiserver. I digged a bit deeper and it looks like kubelet does not sets namespace to be empty and this results in the kubelet trying to post to `/api/<version/events', which a resource that is not exported by the apiserver.
cc


Question: } config.diagnoseMissingEndpoints(eps) <del> framework.Failf("Failed to find expected endpoints:nTries %dnCommand %vnretrieved %vnexpected %vn", maxTries, cmd, eps, expectedEps) </del> <ins> return fmt.Errorf("failed to find expected endpoints, ntries %dnCommand %vnretrieved %vnexpected %v", maxTries, cmd, eps, expectedEps) </ins> } // GetSelfURL executes a curl against the given path via kubectl exec into a

What happened: final passing output: What you expected to happen: Probably if this test passes, it should be showing several real endpoints. I don't know exactly what it does, but im assuming that the output could be clarified a little. How to reproduce it (as minimally and precisely as possible): run and look at the logs. Anything else we need to know?: Environment: Kubernetes version (use ): 1.20 (sonobuoy)
/sig-network
/assign
/triage accepted
^ an interesting case here was that was but there was no SIG (typo in the command that did). should the label be only if there is label on the ticket? /sig network
the PR that adds triage/accepeted automation () was merged 3 hours ago, but it'll "actually" get rolled out in the next prow bump. Mostly when merges. Thanks for flagging though, appreciate it! :)
sorry, missed the actual question! :woman_facepalming: (need moar coffee!! :coffee: ) I don't think we should require a label for adding the label. In case someone comments with the following, we'd still want it to be valid.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
The above test cleans up the By directives and so on, After reading through the test more its clear that every test sets up a scaffold measures a property of that scaffold sometimes deletes something that is recovered in that scaffold (i.e. a pod that is behind a UDP service) So, the thing that was confusing I guess to me is that we validate empty conditions (i.e. no pods available) once we delete a nodePort svc. Meanwhile also noticed that the original work to clean up error reporting needed to cascade over to these tests as well.Have just finished all that, and also updated comments to reflect the Todo around the statistical computation (we have the ability to do extra polling via but we don't actually have any test that leverages this.As a follow on I guess it would be nice if we did have a test that confirmed that pods were load balanced evenly, esp with the new proliferation of different proxy technologies , I.e. a (or something similar


Question: "github.com/go-openapi/spec" "github.com/onsi/ginkgo" <ins> openapiutil "k8s.io/kube-openapi/pkg/util" </ins> "k8s.io/utils/pointer" <ins> "sigs.k8s.io/yaml" </ins> "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions" apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"

Which jobs are failing: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) Which test(s) are failing: [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema [Conformance] Since when has it been failing: 04-26 05:02 PDT Testgrid link: Reason for failure: Apr 26 12:28:01.219: FAIL: failed to explain error running /workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://35.185.126.243 -- explain Command stdout: stderr: the server doesn't have a resource type "ksvc" error: exit status 1 Full Stack Trace () +0x1545 (0xc002ca6100) +0x324 (0xc002ca6100) +0x2b testing.tRunner(0xc002ca6100, 0x4b8d060) +0xc9 created by testing.(T).Run +0x350 [AfterEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] ï¿½[1mSTEPï¿½[0m: Collecting events from namespace "crd-publish-openapi-4377". ï¿½[1mSTEPï¿½[0m: Found 0 events. Apr 26 12:28:01.314: INFO: POD NODE PHASE GRACE CONDITIONS Apr 26 12:28:01.314: INFO: Apr 26 12:28:02.899: INFO: skipping dumping cluster info - cluster too large Apr 26 12:28:02.899: INFO: Waiting up to 3m0s for all (but 50) nodes to be ready ï¿½[1mSTEPï¿½[0m: Destroying namespace "crd-publish-openapi-4377" for this suite. Anything else we need to know: /cc /cc /milestone v1.19 /priority critical-urgent
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

<detailsIn response to : Which jobs are failing: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema [Conformance] 04-26 05:02 PDT Apr 26 12:28:01.219: FAIL: failed to explain error running /workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://35.185.126.243 -- explain Command stdout: the server doesn't have a resource type "ksvc" exit status 1 () (0xc002ca6100) (0xc002ca6100) testing.tRunner(0xc002ca6100, 0x4b8d060) created by testing.(T).Run [AfterEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] ï¿½[1mSTEPï¿½[0m: Collecting events from namespace "crd-publish-openapi-4377". ï¿½[1mSTEPï¿½[0m: Found 0 events. Apr 26 12:28:01.314: INFO: POD NODE PHASE GRACE CONDITIONS Apr 26 12:28:01.314: INFO: Apr 26 12:28:02.899: INFO: skipping dumping cluster info - cluster too large Apr 26 12:28:02.899: INFO: Waiting up to 3m0s for all (but 50) nodes to be ready ï¿½[1mSTEPï¿½[0m: Destroying namespace "crd-publish-openapi-4377" for this suite. /cc /milestone v1.19 </details
/milestone v1.19 Thanks for opening this! I'll get you on milestone maintainers soon :) Also, was taking a look at this as well and noticed this test was recently :
As noted in , this modified an existing conformance test and made it flaky. Please move the new test out of the conformance test and mark it as flaky until this issue has been resolved.

Cc for flake introduced in
It is a bit weird that I cannot reproduce this problem on my local kind cluster by running even after 20 times I suspect this is a cache issue since I once created a crd without short name and forgot to delete it in the test.
/retitle [Flaky Test]: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) /kind flake /remove-kind failing-test
/assign
I can reproduce this problem by the following steps now. It looks like that this problem has something to do with discovery cache. <pre$ cat <summary<detailsapiVersion: kind: CustomResourceDefinition metadata: name: spec: conversion: strategy: None group: crd-publish-openapi-test- names: kind: Service listKind: ServiceList plural: services singular: service shortNames: ["ksvc"] scope: Namespaced versions: name: v1alpha1 schema: openAPIV3Schema: properties: spec: description: Specification of CustomService properties: dummy: description: Dummy property. type: string type: object type: object served: true storage: true </details</summary$ kubectl get crd No resources found in default namespace. $ rm -r $ kubectl explain the server doesn't have a resource type "ksvc" $ kubectl create -f created $ kubectl explain the server doesn't have a resource type "ksvc" $ kubectl explain <summary<detailsKIND: Service VERSION: crd-publish-openapi-test- RESOURCE: spec <ObjectDESCRIPTION: Specification of CustomService FIELDS: dummy <stringDummy property. </details</summary</pre
looks like
/assign


Question: #   DOCKER_MOUNT_ARGS function kube::build::verify_prereqs() { kube::log::status "Verifying Prerequisites...." <ins> kube::build::ensure_tar </ins> if [[ "${1-}" != "clean" ]]; then if [[ -z "$(which docker)" ]]; then

Since I've been having troubles with installation, I've digged a bit and found out that the permissions inside Kubernetes tarballs changed between 0.16.0 and 0.17.0. I've downloaded both tarballs, extracted them into separate directories and then extracted the Kubernetes server tarballs. This is what I get: I suppose, there has been a concious decision to restrict the world's access to those binaries. However, this breaks the installation routine mentioned above and probably makes life a bit more difficult in several other places. I'd be thankful for a correction if this wasn't intended or a pointer to the appropriate commit it this is legit -- I'd correct the installation.
assigning this to you to decide who the eventual assignee should be :)
Any news on this front? 0.20.1 is still restricted.
Apologies for the issue. It wasn't intentional -- likely an artifact of build scripts being run by different people? I'll check out 0.20.2 as I build it to make sure it has the right perms.
Apologies, but 0.20.2 didn't include fixed permissions. I now understand the issue, : "make release" builds a tarball using the permissions on files in your local git tree. I had "umask 027" set when I cloned, so those in-the-repo files had no world-readable perms. Setting "umask 022" before running "make release" is not enough -- it has to be part of creating the local tree when cloning. I'll update instructions and the next release (20.3 or 21.0) should fix this issue. And longer term, we need to put a bit more effort into releasing from a known-good environment and not rely on a developer's workstation's idiosyncrasies -- sorry this bit you.
I can't repro Trying again a different way, but I don't think the umask before clone should do anything (in particular, umasks aren't "sticky" on files, they're sticky to processes.)
And let's pretend on my personal fork isn't woefully out of date (who uses master on their personal fork? I should really just delete that that thing.) Comment updated.
Ah, I see the problem now.

works fine for the server tar, but the full tar also packages examples, etc., and for that, tar tends to just pick up the permissions of the git tree. Got it.
I'll take this. I'm going to write a script to build official bits, and this is easy enough to fix along the way.
Seems not fixed as of Kubernetes 1.0.7
sorry about that. is doing the 1.1.1 release and has been improving release scripts -- is 1.1.1 good re: permissions?
was not built by .
Reassigning to as release-oncall. Quinton, I can mentor or advise as necessary.
And yeah, is fine: There's no real bug here. There was a process issue that occurred with 1.0.7. We can rebuild the binaries if we need to.
FYI, leaving this issue closed, as per 's comment above. If anyone disagrees, please re-open this issue.


Question: // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *DebuggingConfiguration) DeepCopyInto(out *DebuggingConfiguration) { *out = *in <ins> if in.EnableProfiling != nil { in, out := &in.EnableProfiling, &out.EnableProfiling *out = new(bool) **out = **in } if in.EnableContentionProfiling != nil { in, out := &in.EnableContentionProfiling, &out.EnableContentionProfiling *out = new(bool) **out = **in } </ins> return }

What would you like to be : Change the external DebuggingConfiguration member variables to pointer types. Why is this needed: So that we can set their default values to something different from golang's, for example enable profiling by default. /sig api-machinery /cc
The label(s) cannot be applied. These labels are supported: <detailsIn response to : What would you like to be : Change the external DebuggingConfiguration member variables to pointer types. Why is this needed: So that we can set their default values to something different from golang's, for example enable profiling by default. /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
done in