Question: return nil } <del> // Stop stops the gRPC server </del> <ins> // Stop stops the gRPC server. Can be called without a prior Start // and more than once. Not safe to be called concurrently by different // goroutines! </ins> func (m *Stub) Stop() error { <ins> if m.server == nil { return nil } </ins> m.server.Stop() <del> close(m.stop) </del> <ins> m.wg.Wait() m.server = nil close(m.stop) // This prevents re-starting the server. </ins> return m.cleanup() }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm running into a test failure during "make test". To get more output and to avoid issue , I am using: What you expected to happen: The test should pass. How to reproduce it (as minimally and precisely as possible): It happens for me only when running all tests, but then it happens reliably. When limiting the test run with , it works. The failing calls are not surprising, because got unlinked after binding to it. Those might come from 's : it removes everything in the socket dir, which happens to be . For comparison, here's a successful run: Anything else we need to know?: I'm wondering whether it is guaranteed that different tests in the same .go file run serialized. Does anyone know for sure? I'm asking because all tests in use the same socket directory and thus would conflict with each other if they were to run in parallel. Note that the and calls in the failed test are for different pids. I've not checked whether those are truly different processes (as in fork+exec) or worker threads (clone). Environment: Kubernetes version (use ): master branch, commit OS (e.g. from /etc/os-release): Debian Stretch (9.0) Kernel (e.g. ): 4.9.65-3+deb9u2
After fixing the /tmp dir handling with PR I can no longer reproduce this issue. It might be worth keeping this issue open nevertheless, because the issue has not been explained completely and there might be some remaining problem lurking somewhere.
I think you originally wrote this test. Perhaps you can comment here and comment on PR ?
/cc This test has been re-written multiple times, I'll take a look ASAP :)
/sig node
Will take a look ASAP.
I experienced the same. What's interesting is I can get it to pass when running the devicemanager suite, but only about half the time.
/area hw-accelerators
Thanks a lot for reporting the issue! I think your analysis is right. Looks like the problem is because the devicemanager unit tests are interfering with each other. I will make a change to have each unit test in the file to use a separate test tmp dir.
I'm not exactly sure why, though.

Do they really run in parallel? According to my understanding of testing.Parallel, one has to explicitly request parallel execution, and the tests don't do that. When they run sequentially, then it isn't obvious why the tests interfere and thus using per-test socket dirs would be just a band-aid. Please have a look at PR , it has such a change.
Ah sorry I didn't realize you were already working on a fix. I think go test by default run tests in parallel, so we need to make sure each unit test is self contained. That also explains why you hit this problem more easily when running more tests. I think we perhaps need to modify setup() to make sure each newly created manager uses a separate test dir. Maybe you can take a look at my WIP change and see whether you can merge some of its changes into PR ?
I suggest you take a look at the documentation: On Fri, 9 Feb 2018, 22:34 Jiaying Zhang, <:
On Sat, 2018-02-10 at 05:31 +0000, Vikas Choudhary (vikasc) wrote: True, but which unit tests run concurrently? To answer that, I re-read the "testing" documentation and ran a quick experiment. That confirmed that individual tests inside the same test binary run sequentially. A test function has to explicitly call t.Parallel() before the next test starts, and that test blocks unless it also calls t.Parallel(). In other words, using the same /tmp/deviceplugin between all test functions should have been okay. I suspect it goes wrong sometimes because one test function starts a goroutine, does not wait for its completion, and that goroutine then interferes with the next test function when using the same dir, depending on timing. Running other test binaries in parallel and thus increasing system load probably also matters, together with overall number of CPUs. I think merging PR makes sense because a dynamic test directory is more flexible and enables running the tests in parallel (which could be a separate PR or be done as part of PR ), but I also think that it would be good if someone with a better understanding of the code that gets tested here could look into this race condition. [(quite) a bit later...] I chased down the spurious unlinkat() to UnixStream.Close for the socket, which unlinks the socket path [1].

By wrapping the listen socket such that a custom Close logs the call and a stack trace, I now have the smoking gun. [1] With some extra logging one can see that the that was created for TestNewManagerImplStart gets closed and unlinked while TestDevicePluginReRegistration already runs. The goroutine which runs too long is the from was called, but didn't wait for the goroutine to actually stop: That now begs the question: what is the right way to stop a grpc server? Below is the change I made to to track when it closes the socket:
On Tue, 2018-02-13 at 15:57 +0100, Patrick Ohly wrote: Server.Stop is the right way, it just doesn't fully stop the server. I consider this an issue in gRPC, therefore I have created https://github .com/grpc/grpc-go/issues/1861
Is it possible that defer statements are executed at the same time another test might start? EDIT: NVM looks like that's not the case
On Tue, 2018-02-13 at 07:39 -0800, Renaud Gaubert wrote: No, defer statements are guaranteed to run before a function returns. I think they can even change the return code and/or catch a panic.
thanks so much for your detailed debugging analysis and details about the tools you used (e.g. using filtered strace). I'm fairly new to golang and I've been messing around with the k8s source locally trying to get a feel for the project and work towards contributing, and I was debugging this test on my own but flailing against how to instrument the running tests. very much appreciated and nice work!


Question: cadvisortest "k8s.io/kubernetes/pkg/kubelet/cadvisor/testing" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" containertest "k8s.io/kubernetes/pkg/kubelet/container/testing" <ins> kubecontainertest "k8s.io/kubernetes/pkg/kubelet/container/testing" "k8s.io/kubernetes/pkg/kubelet/kuberuntime" </ins> "k8s.io/kubernetes/pkg/kubelet/leaky" serverstats "k8s.io/kubernetes/pkg/kubelet/server/stats" statustest "k8s.io/kubernetes/pkg/kubelet/status/testing" <ins> "k8s.io/kubernetes/pkg/volume" </ins> ) func TestFilterTerminatedContainerInfoAndAssembleByPodCgroupKey(t *testing.T) {

Rotated container logs space usage doesnt seem to be counted towards the epehemral storage limit. Always only current log file stats are taken into account. rotated log files usage also should be counted towards ephemeral-storage usage No response <detailsTested on Kubernetes Version : 1.21.7, 1.22 Container Runtime : Cri-o </details<details</details<details</details<details</details<details</details<details</details
/sig node
/cc
/assign we can see if there is a issue
Were the rotated log files in /var/log/pods generated by yourself(not by k8s)? The cadvisor can only collect log files on container's filesystem.
No, I have not created any files. they are generated by k8s itself.
Okay, I see. As the cri-o runtime uses cadvisor stats provider to get container stats, it caculates container logUsage by formular . but the default root dir or log dir of cri-o is not /var/log/pods/{podName}/, so theses log files are not on container's filesystem, then cadvisor may not caculate the data accurately. /cc the containerd uses cri stats provider and gets logstats by HostStatsProvider, it can get right data.
In the kubelet code, , the log path is in the format , but the directory of your environment is , I did not see , Did you set this path yourself?
sorry I truncated the name for better readability, the actual path name is . So I suppose its taking in the above format only.
log files gets generated in the path /var/log/pods/defaultephem15334a37d-df02-435c-b385-/con2 for cri-o in my setup
/area cadvisor /triage accepted /priority important-longterm
/retitle Rotated container log files size not counted towards ephemeral-storage's limit on CRI-O
hi, In the past two days, I have studied the collection of file information by cAdvisor. docker and cri-o runtime basically have the same mechanism in cAdvisor. I have verified that when the runtime is cri-o, the log information statistics are also correct. So I need you to provide more detailed information. I found that the command you show is: , please help to confirm whether the "logs" information belongs to the container con2.

or use the command to try
I have verified and info provided is for con2 only . I have the sample yaml used for testing , please let me know if I'm doing it right or anything missing. I have set the kubelet log configuration as containerLogMaxSize: "5Ki", containerLogMaxFiles: 5 After applying the above yaml stats summary shows con1 logs usedbytes always 0 , con2 logs usedbytes as 4096
attached the stats summary of the command And I'm testing on kubernetes 1.22 release
/assign
what version of CRI-O/kubernetes are you using? using latest main of both kubernetes and cri-o with your reproducer causes the container to be evicted because of it exceeding ephemeral storage
oh I see now: my mistake
another mistake: I was using CRI stats provider, instead of cAdvisor stats provider (as is default)
Is CRI stats provider used by default in 1.23 by default? And also problem is seen only if containerLogMaxSize is less that the ephemeral storage limit set , if its more than the ephemeral storage limit , its the right behavior , pod gets evicted in my setup.
at long last, I believe I've found the issue!


Question: package proxy import ( <ins> "bufio" </ins> "bytes" "context" "fmt"

(9.8, critical) With a specially crafted request, users that are authorized to establish a connection through the Kubernetes API server to a backend server can then send arbitrary requests over the same connection directly to that backend, authenticated with the Kubernetes API server’s TLS credentials used to establish the backend connection. Thanks to Darren Shepherd for reporting this problem. CVE-2018- is fixed in the following Kubernetes releases: Kubernetes API server Kubernetes v1.0.x-1.9.x Kubernetes v1.10.0-1.10.10 (fixed in ) Kubernetes v1.11.0-1.11.4 (fixed in ) Kubernetes v1.12.0-1.12.2 (fixed in ) Note: If you are using binaries or packages provided by a distributor (not the ones provided in the open source release artifacts), you should contact them to determine what versions resolve this CVE. Distributors may choose to provide support for older releases beyond the ones maintained by the open source project. Clusters = 1.6.x that run aggregated API servers (like the metrics server) that are directly accessible from the Kubernetes API server’s network. If there are aggregated API servers configured in a cluster, the following command will return the names of the associated APIService objects (if no names are listed, or the kube-apiserver is an older version that does not have the apiservices API, then the cluster has no aggregated API servers configured): Clusters = 1.0.x that grant pod exec/attach/portforward permissions to users that are not expected to have full access to kubelet APIs An API call to any aggregated API server endpoint can be escalated to perform any API request against that aggregated API server, as long as that aggregated API server is directly accessible from the Kubernetes API server’s network. allows all users (authenticated and unauthenticated) to perform discovery API calls that allow this escalation against any aggregated API servers configured in the cluster. A pod exec/attach/portforward API call can be escalated to perform any API request against the kubelet API on the node specified in the pod spec (e.g. listing all pods on the node, running arbitrary commands inside those pods, and obtaining the command output). Pod exec/attach/portforward permissions are included in the admin/edit RBAC roles intended for namespace-constrained users. This section lists possible mitigations to use prior to upgrading.

Note that many of the mitigations are likely to be disruptive, and upgrading to a fixed version is strongly recommended. suspend use of aggregated API servers (note that this will disrupt users of the APIs provided by the aggregated server) disable anonymous requests by passing to the kube-apiserver (note that this may disrupt load balancer or kubelet health checks of the kube-apiserver, and breaks setup flows) remove all anonymous access to all aggregated APIs (including discovery permissions granted by the ) suspend use of aggregated API servers (note that this will disrupt users of the APIs provided by the aggregated server) remove all access to all aggregated APIs (including discovery permissions granted by the ) from users that should not have full access to the aggregated APIs (note that this may disrupt users and controllers that make use of discovery information to map API types to URLs) Remove pod exec/attach/portforward permissions from users that should not have full access to the kubelet API There is no simple way to detect whether this vulnerability has been used. Because the unauthorized requests are made over an established connection, they do not appear in the Kubernetes API server audit logs or server log. The requests do appear in the kubelet or aggregated API server logs, but are indistinguishable from correctly authorized and proxied requests via the Kubernetes API server.
, why doesn't kubelet's webhook authorization help here? Shouldn't it reject subsequent requests on still open TCP connection from APIServer?
Because the kube-apiserver <-kubelet connection was established with the kube-apiserver's TLS credentials, which are broadly authorized against the kubelet API. The kubelet would authorize the kube-apiserver to make that request, and allow it.
Hmm, I misunderstood how webhook authorization works then, I thought that no requests are trusted in that mode and even for /proxy requests, initiator's credentials are passed by APIServer to kubelet, which then comes back to apiserver for SubjectAccessReview. So I thought scheme is like following: Initiator -APIServer (authn/authz) -kubelet -APIServer (SubjectAccessReview) But seems it is not the case :(
the credentials authorized by the kubelet in that case are the apiserver's
Thanks for the fix and the detailed summary! Just for clarification - this vulnerability can't be exploited by anonymous users from outside of the cluster, as long as we disable , right?

I'm asking because my users tend to open up K8S API endpoint to the Internet, occasionally but not always restricting accesses to the endpoint at L4 by the source IP.
Correct, escalations start with a single authorized request through kube-apiserver to the backend. Preventing that (either by disallowing anonymous requests with --anonymous-auth=false or by removing the default discovery permissions granted to anonymous users) mitigate the anonymous escalation. Be aware that can disrupt normal operations of a cluster, including load balancer and kubelet detection of kube-apiserver health (via anonymous requests to /healthz), and kubeadm join flows that depend on public access to a config map containing cluster information in the namespace.
Initiator -APIServer (authn/authz) -kubelet -APIServer (SubjectAccessReview) Interesting, are there any drawbacks of passing initiator's credentials all the way to kubelet? Would it allow to eliminate one more "superuser" in the form of APIServer?
Exposing initiator's credentials broadly to the cluster is a non-goal. Additionally, many API callers have non-forwardable credentials (TLS client certificates). The apiserver could forward information about the initiator to the kubelet, but that information would have to be trusted (e.g. initiator info in headers, verified by mTLS connection proving the apiserver provided the info). Interestingly, that is exactly the scenario exercised by kube-apiserver -aggregated API server connections, where the kube-apiserver acts as an authenticating proxy, and the aggregated API server trusts the identity assertions sent to it by the kube-apiserver. not really, as long as kube-apiserver is responsible for terminating and asserting authentication information to backends.
- I'm having a hard time understanding the full extent of this. If the API is in a private network (no public exposure), with RBAC enabled, but still around, would it still be vulnerable to this?
The first step in the escalation is a properly authorized request to the kube-apiserver that gets proxied to a backend server. A malicious request must be able to reach the kube-apiserver for the vulnerability to be exploited.
That's what I thought, thanks so much for taking the time to answer so quickly!
Thanks for the clarification !
Hi, I want to know why the issue isn't fixed in 1.9.x?

We have a lot of clusters running the 1.9.x k8s.
Pardon stupid question, but is Heapster considered aggregated API server same as metrics server?
The Kubernetes project supports the most recent three minor releases with bug and security fixes. The currently supported releases are 1.10.x, 1.11.x, and 1.12.x. If you are running builds provided by a distributor (not the binaries or packages provided by the Kubernetes project), contact their support to determine which of their versions are vulnerable. Some distributors may apply fixes to versions prior to the open-source-supported versions. No, Heapster is not an aggregated API server. Aggregated API servers register objects with non-nil fields, and requests to endpoints on the kube-apiserver are proxied to the appropriate aggregated server.
Are there any detail procedures to reproduce the issue? So we can verify that if we have fixed it on 1.9.x. Thanks a lot.
How to reproduce the issue, so we can check if the kubernetes version we're using is affected by this issue? An example script/command shows how gets all pods information on that node can be helpful.
Is it enough to just update the API Server to a fixed version, or should all the components (scheduler, kubelet, etc.) be updated to the fixed Kubernetes version to be secure?
, fix is in the apiserver codebase, so it should be enough to update just it
Does this affect ELB/ALB/Ingress controllers in any way? I mean if K8s API is hidden in a private network but load balancers for services (not for K8s API) are wide open, is it possible to use this vulnerability?
If the request cannot reach the API servers, the exploit cannot work. Even if the request can reach the API servers, but not have the permission to establish backend/proxy connection, it is also fine. But if a user in your cluster can use kubectl "freely", it means that they can connect to K8s API server, and the CVE may be applied here. Or, if the services that run on your cluster has a role that allows pod exec/attach/port-forward, it can proxy the request from the service to the API server.

Or if one of your services are API server extension, and your users can interact with it directly, then he/she can craft the exploiting request
Could someone give an example of an aggregated API endpoint that is usually enabled by default?
the metrics server is an aggregated API server, so is often a valid endpoint
What is the safest way to "suspend use of aggregated API servers" and to "remove all anonymous access to all aggregated APIs" () ? Is it sufficient to move aside the proxy-client-key-file mentioned here: ?
Don't laugh at this question. is Kubernetes version 1.7.11 affected? I would assume so. But if anyone can confirm this will help me out considerably. Thanks James Sorry I won't add another comment, I'll edit this one to thank you for your reply below, I mis-read the post on affected versions, I did not see 1.0.x-1.9.x, Sorry for wasting your time with the question.
I would say Yes, as Kubernetes v1.0.x-1.9.x 1.7.x falls within that range.
Sorry to bug you, but it would be a huge help to get an answer to my questions in .
I know the official docs on upgrading state to to from 1.7 to 1.8 then to 1.9 and then to 1.10, Would anyone know of a way to from 1.7.11 to 1.10.11. If not we will just figure out the steps and do the 3 separate upgrades. If something similar has been done before we'd like to hear about it as we have a few customers on 1.7.11 and all need to get upgraded. We will of course do our own trial and error in a lab and try and script much of it.
I don't understand why this leads to security issues. what is the "In default configurations"?
Without knowledge of your particular cluster and use, that is difficult to say. The safest way to suspend use of aggregated API servers might be to leave their objects registered, and scale down the deployments backing those objects, so that no aggregated server instances are available to receive requests.

Some potential disruptions that would cause: If portions of the cluster depend on an aggregated server (like HPA depending on metrics-server), this would break those dependencies aggregated servers that cannot be reached cause the garbage collector and namespace cleanup controllers to wait to perform some operations until the unavailable server can be reached again. This could prevent deletion of namespaces, and prevent foreground deletion of objects until the servers are made available again Removing anonymous access would either consist of disabling anonymous requests entirely with (see issue description for possible problems with that approach) or removing authorization permissions from anonymous users (specifically the user and group). If using , that includes removing the anonymous subjects from the and on restart. Keep in mind that preventing anonymous access still leaves the cluster vulnerable to escalation by authenticated users. Moving the would just prevent your kube-apiserver from starting Skip-version upgrading is not supported. A low-privilege request (like an anonymous call to a discovery API listing available resource types, like ) can be escalated to make any API call (including reading and writing API objects) to the aggregated server (like ). The default configuration refers to the discovery permissions granted to all users by .
on a 1.9.5 version i have it as --anonymous-auth=false and i deny all unauthenticated requests
In our Kubernetes cluster (Azure ACS, Kubernetes version# 1.8.4), we are using kubeconfig in order to establish proxy connection with kubernetes cluster. This is limited to only certain operations team members. It means in order to set connection with kubernetes api server, certificate from kubeconfig is used. Also, all the apis hosted on this cluster are exposed via azure api management service. In that case, is our kubernetes cluster still applicable to above vulnerability?
you mentioned "Aggregated API servers register APIService objects with non-nil spec.service fields" ... so is this a good query to see if someone has one of those in their environment?
This would work without jq: Added to the description
Gravitational has created a vulnerability test utility:
Under "affected configurations", it seems to suggest that if you run the given kubectl command and have no output, you do not have any exploitable endpoints for the aggregation api portion of this vulnerability. However, in "vulnerability impact" immediately below it states "In default configurations, all users (authenticated and unauthenticated) are allowed to perform discovery API calls that allow this escalation.".

Does this actually more precisely mean that the ability to discover vulnerable api calls that might exist is enough access to perform the exploit, if you happen to have exploitable endpoints? Asking because we appear to have no output from the kubectl command, but I can anonymously call "get --raw /apis" with . Unsure if this cluster is currently exploitable or not via this mechanism.
- clarified the "Affected configurations" and "Vulnerability impact" sections
thanks
Great thanks to !
there are two tests circulating for configurations that are vulnerable to the aggregated API exploits: the test you to the issue description, and which posted above. As mentioned in this issue: their results aren't always the same. In fact, we have clusters that are not affected according to your test but are affected according to . Could you clarify whether a negative from your test plus a positive from Gravitational's test indicates that we are not exposed?
the gravitational tests for ... however, if there are NO aggregated apis (as shown by 's command) then the cluster is still ok (hopefully can confirm this). You are still vulnerable to the other code path so you will have to review the RBAC permissions.
If there are no objects that route to a service (the check), that means no aggregated servers are configured in the cluster, and the anonymous escalation via aggregated discovery API requests is not possible. That said, if you are still running a vulnerable version and later add an aggregated API server, it would become exploitable.
Thanks very much for the clarity.
What about Kubernetes <1.5.x that no have "apiservices" daemon running/installed? And Could you please explain "user", which user do you mean. User that have shell access without root-access to kube-masters (what i think) and/or user from outside through an hosted application (ex. apache server on running pod-API-Server) and/or user that have general access to kube machines but without shell access. Thanks in advance
The APIService API was alpha in 1.6.x, and beta and enabled by default in 1.7.x+. kube-apiserver releases without an APIService API do not have the ability to proxy to aggregated API servers, so anonymous escalation via aggregated discovery API requests is not possible on those versions. Any actor that can send a network request to the kube-apiserver.

Escalation paths for this vulnerability begin with a network request to the kube-apiserver, which must first pass authorization, then be proxied to a backend (either to a kubelet in the case of pod exec/attach/portforward, or to an aggregated API server in the case of aggregated APIs).
The pod exec/attach/portforward escalation path applies back to 1.0.x
Fixed in for
Updated the description with a link to the document and recording of the post-mortem for this incident. Links to action items are in the document and reference this issue.
/label official-cve-feed (Related to )


Question: ], importmap = "k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/errors", importpath = "k8s.io/apimachinery/pkg/util/errors", <ins> deps = ["//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library"], </ins> ) filegroup(

What happened: I configured my API server with a Webhook token authentication server by way of the "authentication-token-webhook-config-file" command-line flag, and confirmed that the API server does pass bearer tokens through, issuing HTTP POST requests with a JSON-encoded body. When my Webhook server indicates authentication failure, it responds with HTTP status code 200 and a JSON-encoded in the body, this time with the object's "Error" field populated. Here's an example: When the API server receives this response, it refuses the original motivating request with an HTTP status code 401 response, but the API server writes the following into its log file: This use of this default error message is due to an omission in file , where , and . What you expected to happen: The API server should instead log a message that includes the value from . Per a brief discussion with , he agreed that the API server log message should include such an error message from the Webhook server, provided it's present and nonempty. How to reproduce it (as minimally and precisely as possible): Configure an API server to delegate to a Webhook token authentication server. Force that Webhook server to respond "successfully" with an HTTP 200 response, but with the "Authenticated" field set to false (or omitted) and with a particular error message in the "Error" field. Make an HTTP request of the API server with an HTTP "Authentication" header with a value starting with the string "Bearer " followed by at least one more character. Inspect the logs from the API server, looking for a message that begins with "Unable to authenticate the request due to an error." Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kubeadm_
/sig auth
/assign


Question: framework.ConformanceIt("should adopt matching pods on creation and release no longer matching pods", func() { testRSAdoptMatchingAndReleaseNotMatching(f) }) <ins> ginkgo.It("Replicaset should have a working scale subresource", func() { testRSScaleSubresources(f) }) </ins> }) // A basic test to check the deployment of an image using a ReplicaSet. The

[x] two weeks soak end date : 26 Feb 2021 According to this APIsnoop query, there are still some remaining RESOURCENAME endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a ReplicaSet with a static label Read the ReplicaSetScale Update the ReplicaSetScale Get the ReplicaSetScale to ensure it is updated Patch the ReplicaSetScale Read the ReplicaSetScale to ensure it is Patch Delete the Namespace and ReplicaSet Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. example endpoint | useragent ----------------------------------------|
patchAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource readAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource replaceAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource (3 rows) If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture
/triage accepted
Presented in Conformance meeting of 9 Feb 2021 and it was agreed that that method see to achieve the purpose and a PR can be created. A similar approach would be taken for the DepoymentScale endpoints.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: openssh-client  nfs-common  socat  <ins> udev  </ins> util-linux COPY cni-bin/bin /opt/cni/bin

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Cloud providers are using the command-line tool to retrieve status and format volume mounts. is missing from the hyperkube image resulting in missing volumes. Issues: What you expected to happen: Volume mounts on cloud providers are expected to be formatted and mounted. How to reproduce it (as minimally and precisely as possible): GCE and Openstack volume mounts are effected. Anything else we need to know?: Environment: Kubernetes version (use ): 1.9, 1.10+ Cloud provider or hardware configuration: GCE and Openstack OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/cc
/sig storage

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): container-os Kernel (e.g. ): Linux gke-wordpress-cluster-default-pool--m764 4.4.21+ SMP Fri Feb 17 15:34:45 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux Install tools: Others: What happened: ` I was able to bring up wordpress okay the first time, except GKE wasn't creating loadbalancer ips due to a quota issue, which I resolved, note at this point the mysql pod was up and had attached to it's volume. Upon deleting the wordpress deployment and creating it again I started getting the above errors. I deleted the mysql pod as well and brought it up again to see that it had the same issue. The volumes are backed by a gluster cluster on GCE. Looking at the brick logs on one of the gluster nodes I see I've tried restarting kubelet on the node. I can't find the kubelet log file on any of gke nodes. And I don't know how to get the kube-controller and apiserver logs from the master (GKE). I suspect it's a failing of the glusterfs client on the gke nodes? What you expected to happen: Deployment to mount the volumes successfully. How to reproduce it (as minimally and precisely as possible): Not sure how, but I've run into this intermittently Anything else we need to know: The volumes are backed by gluster
/cc
If you delete pod and create pods soon, it is possible that you could see some errors like "timeout expired waiting for volumes to attach/mount" temporally. Could you wait for a few more minutes to see whether the pods could be started correctly?
I actually tried that, deleted the pods, then waited 5 minutes and recreated them with the same error. Maybe it wasn't long enough
Can you post output of ? It may have some other events for failure..
I deleted the cluster this afternoon, but I checked constantly and those were the only errors, Timeout on mount and a failed to sync error
If you use level 4 log, it should also give the detailed messages of failed mount so that we can figure out why it failed.

Can you email me your project name, cluster name, and zone information? I can take a look at the master logs to see if there's anything useful there.
This is starting to happen more frequently..
could you provide more details about in what situation it happened frequently? Kubelet log could give us more information. If you use GKE cluster, run "journctlctl -u kubelet /tmp/log" to get the log. You could also share with us the log if it is ok.
This seems to be the relevant part of the log I re-created that and my pods sprang back up. Weird thing is I had created that, along with the pvcs and pvs but seems it's somehow getting deleted. There's almost certainly certain I'm doing wrong but I'm not sure what. ]
HI Jonathan, can you clarify what you had to recreate?
The glusterfs endpoints , Michelle Au <:
I have a very similar problem running OpenShift 3.4 in AWS EC2 doing the same exact thing. Brand new OSE install [root master]# oc version oc v3.4.1.12 kubernetes v1.4.0+ features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.31.40.126:8443 openshift v3.4.1.12 kubernetes v1.4.0+ POD in question is the docker-registry (kind of important for OSE to work) I've followed this to see if it would fix the issue: But no luck or change. Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] then: Unable to mount volumes for pod "docker-registry-3-afbbt_default(-1a45-11e7-abf8-)": timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] I have no idea what log(s) to look at, this was of no help when I deployed a new docker-registry: oc logs -f dc/docker-registry Any help or guidance would be greatly appreciated as this is a blocker issue for me.

thanks and best wishes,
are you using glusterfs for registry-storage volume or other type of volume? Could you share us your kubelet log if possible over slack Thanks!
Hi Jing Xu, Would you know where this log would be? Or is the name kublet<somethingso I may find it for you? I just requested to join the slack channel, as soon as I'm signed in I will share the log. Probably tomorrow as I'm on the east coast USA Thank you! Joel
For you are using GKE with GCI image, you can use "journalctl -u kubelet " to get the log. Otherwise, it should be in " , shepp67 <:
I also experienced this. Kubernetes version:Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Install tools: custom That kubernetes worker vm actually had volumes attached. I saw it on OpenStack dashboard. But: and actually: Besides, this is a new worker vm. I have other worker vms already running, using cinder volumes and there were no such errors. I use Helm to deploy this Pod. I have deployed it earlier and removed it. But it was 2 days ago. I found the solution here: . On that worker vm I run: Edit: or upgrade Kubernetes to = 1.6.0-beta.3 I don't know if this is the solution to issue.
I'm seeing this same error on all my manually launched private Ubuntu based clusters (6), running 1.5.3 and 1.6.0. In my case the messages reference only stopped pods launched by CronJob Jobs. The messages indicate issues mounting every volume in the pod including PVs and node-local volumes. These pods, and future ones launched by these CronJobs, are fully successful when they run. It is only after they run to successful completion and exit that these messages show up. Glancing at the kubelet code, this appears to be coming out of some housekeeping/synch loop.
I've identified a bug and am working on a fix for when "Unable to mount volumes for pod..." + "Error syncing pod" happen indefinitely BUT the pod is actually running just fine, as observed by .

For others, if you see this message but your pod is Running, you can safely ignore it. It's caused by a small limitation of the reconstruction of state (what volumes are mounted) so there is a chance you will see this only if you restart kubelet while your pods had volumes mounted. correction: you say they even Completed, they weren't just Running fine. So yeah, it's just misleading logging and not really an indication of anything breaking
Because the error message is very generic, I think there might be different problems mentioned in this thread. 's issue could be fixed by 's PR are you still have any issues with mount? could you also let us know whether you still have the issue? Thanks! Jing
Hi my issue was related to which helped me sort out. I was creating the gluster endpoints without a service and turns out the kube-controller deletes all such endpoints that are not attached to a service at regular intervals. So after every so often when a pod went down and tried to come up again and mount its respective pv it failed because it couldn't connect to gluster. The fix is to create a service for the gluster endpoints.
I also just ran into such timeout issue: After running a load test against my postgres backed web application, the database didn't respond anymore for still unknown reasons. According to kubernetes it was up and running The log showed a bunch of these messages. Not sure yet why but possibly it couldn't access the volume anymore: Short after that, the pod entered "error" state: The event log showed: This error disappeared after a few minutes (not sure how long exactly, but it took a while), but then I got the same error as described here: I couldn't figure out anything more on my own. Nothing in the syslog, nothing more than this in the kubelet log. Finally I rebooted the machine which fixed the issue.
cc do you know what error of the network not ready? It might be helpful if you could share the kubelet log with us to debug. Thanks!
AFAIK, the "network not ready error" is benign, kubelet will retry after this error () cc
I hope I didn't miss some secrets in there, quite a talky log. But here it is: The database didn't respond anymore because it ran OOM and was killed, as also shown in the log.

Although it appears like it was OOM killed after it became unresponsive in the log.. Either way, a bunch of other things got killed too, so granted - "not ideal" situation :). Yet it shouldn't drop the volume and should re-attach it.
I think you are hitting the issue mentioned above. He has a PR to fix it. I will make sure it can merge soon. Thanks!
Great, thanks! Do you know roughly when this will be available in GCE?
you mean GKE or GCE? For GKE it will take a little bit longer. On GCE, when the PR gets merged, the fix should be available when the next release is out (I think should be soon)
Running on GKE. I'd rather not launch my product with this bug around. Is there a workaround for it or something I can monitor for? Or maybe it could get hot fixed on GKE? If there is no workaround, this effectively renders GKE useless for stateful stuff.
It's hard to say if volumes are actually being dropped/unmounted or if the FailedMount errors are wrong & misleading, so I wouldn't go so far as to say it's broken because of some volume issues. It could be something else entirely. If nothing else, the PR jing's referring to will clean up the event noise and help us debug the root cause. In the meantime, I wouldn't trust the FailedMount events to be accurate, so if your applications are unresponsive, there may be other causes worth considering
Well, I only can say that my volumes were dropped and it didn't recover on it's own (only after reboot). Dropped a seen by kubernetes, dunno if the volumes were still attached but they weren't usable by my scheduled database. To be clear: I'm not "complaining" and this was just test/R&D, but it would this is the scariest kind of issue for me to happen in prod, given that it's a managed service where I can't find a workaround on my own. Rebooting fixed it in my case, but that's a big hammer..
You don't need to reboot the node. Delete the pod and restart it should work for this issue. Is this workaround ok for your case? , discordianfish <:
Good to know, thanks!

Will try this next time.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hi, I'm facing the same issue with k8s 1.9, installed by kops on AWS. volume created successfully on AWS and attached to the right node, but the pod is failed to start.
/remove-lifecycle stale
could you please provide more details about your issue? What are the steps to reproduce it? Thanks!
yeah can you open a new bug for EBS and cc me with details? I was hoping we fixed most of EBS problems in 1.9 - I will be more than happy to walk through and find what is causing volume to not mount. As I understand this is different from typical EBS problems where volume fails to attach. In your case, volume is being attached to right node but it is not being seen by kubelet.
I found the issue, and it was the fact that i used c5 instances with nvme, this is why the volume never been attached as k8s looking for /dev/xvdXX and with nvme that path is different. I did notice that in 1.9 k8s should support nvme natively but it seems that it's still not stable. thanks
oh weird. You should open a bug about NVMe support then. I was thinking it would just work.
cc Is there an issue related to nvme support? please let us know any issues you still have with 1.9. Thanks!
This is nvme on AWS, not related to the work is doing on gce.
Thanks Definitely related to NVMe support. Ran into this trying to mount a gp2 EBS volume on a node which was an m5 EC2 instance.
I think I am seeing like this with , but unsure of "why" at the moment. It would seem works perfectly fine.
did you guys had a chance to open a BUG for NVMe support on EBS? If you do - please paste output of and PersistentVolume spec.

We can't fix it unless we know bit more. :(
sorry, we worked around this by reverting back to m4 class instances. It should be simple enough to reproduce however... create a k8s cluster w/ m5 EBS backed nodes and create a simple pod w/ a persistent volume. You'll notice that the pod will never launch and you should be able to debug further. If I do run into this again I'll be sure to get some more data for you.
I too have the error, error syncing pod. kubectl describe pod kafka1-kafka-0 .... .... .... Warning FailedSync 5m (x141 over 35m) kubelet, 10.186.. Error syncing pod Warning BackOff 57s (x164 over 35m) kubelet, 10.186.. Back-off restarting failed container kubectl logs kafka1-kafka-0 -p export KAFKABROKERID=kafka1-kafka-0 export KAFKAADVERTISEDLISTENERS=PLAINTEXT://172.30.*.:9092 /etc/confluent/docker/run sh: 3: /etc/confluent/docker/run: not found


Question: name = "go_default_library", srcs = ["flock_unix.go"], tags = ["automanaged"], <del> deps = ["//vendor:golang.org/x/sys/unix"], </del> ) filegroup(

BUG REPORT: Kubernetes version latest master: Environment: Ubuntu 16.04 LTS, Linux 4.2.0-42-generic: What happened: The doesn't provide mutual exclusion. Two processes can both grab the same flock at the same time. What you expected to happen: Only one process should be able to grab the flock. How to reproduce it (as minimally and precisely as possible): Anything else we need to know: Although the flock doesn't work, the does work. Since we usually specify together with , so the 1st process will exit anyway. However we still need to fix the flock..
cc
Sent a PR to fix this ()


Question: go_test( name = "go_default_test", <del> srcs = ["insecure_serving_test.go"], </del> <ins> srcs = [ "insecure_serving_test.go", "options_test.go", ], </ins> embed = [":go_default_library"], deps = [ "//cmd/controller-manager/app/options:go_default_library",

/kind bug /sig scheduling What happened: kube-scheduler that is pointed to a componentconfig file having ignores that it should use this kubeconfig file. Instead, it uses the in-cluster config (as is neither provided). What you expected to happen: The kube-scheduler should respect the kubeconfig provided in the . How to reproduce it (as minimally and precisely as possible): Starting image with whereby the content of looks like this: makes the scheduler logging The exact same configuration as above works with branch. Seems like has introduced the behaviour. The kube clients are created but the componentconfig file is loaded . /cc Environment: Kubernetes version (use ): 1.11.0-rc.2
/milestone v1.11 for triage
Could you please take a look at this? It seems related to the changes you made recently.
/assign
/assign /priority critical-urgent bumping priority since this would be a 1.11 regression
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: pv:            testVolumeWithNodeAffinity(t, &v1.VolumeNodeAffinity{}), }, { <del> name:          "valid-constraints", </del> <ins> name:          "select-nothing", expectSuccess: false, pv:            testVolumeWithNodeAffinity(t, &v1.VolumeNodeAffinity{Required: &v1.NodeSelector{}}), }, { name:          "select-nothing-empty-terms", expectSuccess: false, pv: testVolumeWithNodeAffinity(t, &v1.VolumeNodeAffinity{ Required: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{}, }, }, }, }), }, { name:          "valid-multiple-terms", expectSuccess: true, pv: testVolumeWithNodeAffinity(t, &v1.VolumeNodeAffinity{ Required: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{ { Key:      "test-key3", Operator: v1.NodeSelectorOpIn, Values:   []string{"test-value1", "test-value3"}, }, }, }, { MatchExpressions: []v1.NodeSelectorRequirement{ { Key:      "test-key2", Operator: v1.NodeSelectorOpIn, Values:   []string{"test-value0", "test-value2"}, }, }, }, }, }, }), }, { name:          "valid-multiple-match-expressions", </ins> expectSuccess: true, pv: testVolumeWithNodeAffinity(t, &v1.VolumeNodeAffinity{ Required: &v1.NodeSelector{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: The terms are ANDed, which is not correct:
cc


Question: } } <ins> func TestAssignsDefaultServiceAccountAndBoundTokenWithNoSecretTokens(t *testing.T) { ns := "myns" admit := NewServiceAccount() informerFactory := informers.NewSharedInformerFactory(nil, controller.NoResyncPeriodFunc()) admit.SetExternalKubeInformerFactory(informerFactory) admit.MountServiceAccountToken = true admit.RequireAPIToken = true admit.boundServiceAccountTokenVolume = true // Add the default service account for the ns into the cache informerFactory.Core().V1().ServiceAccounts().Informer().GetStore().Add(&corev1.ServiceAccount{ ObjectMeta: metav1.ObjectMeta{ Name:      DefaultServiceAccountName, Namespace: ns, }, }) pod := &api.Pod{ Spec: api.PodSpec{ Containers: []api.Container{{}}, }, } attrs := admission.NewAttributesRecord(pod, nil, api.Kind("Pod").WithVersion("version"), ns, "myname", api.Resource("pods").WithVersion("version"), "", admission.Create, &metav1.CreateOptions{}, false, nil) err := admissiontesting.WithReinvocationTesting(t, admit).Admit(context.TODO(), attrs, nil) if err != nil { t.Fatalf("Expected success, got: %v", err) } expectedVolumes := []api.Volume{{ Name: "cleared", VolumeSource: api.VolumeSource{ Projected: &api.ProjectedVolumeSource{ Sources: []api.VolumeProjection{ {ServiceAccountToken: &api.ServiceAccountTokenProjection{ExpirationSeconds: 3600, Path: "token"}}, {ConfigMap: &api.ConfigMapProjection{LocalObjectReference: api.LocalObjectReference{Name: "kube-root-ca.crt"}, Items: []api.KeyToPath{{Key: "ca.crt", Path: "ca.crt"}}}}, {DownwardAPI: &api.DownwardAPIProjection{Items: []api.DownwardAPIVolumeFile{{Path: "namespace", FieldRef: &api.ObjectFieldSelector{APIVersion: "v1", FieldPath: "metadata.namespace"}}}}}, }, }, }, }} expectedVolumeMounts := []api.VolumeMount{{ Name:      "cleared", ReadOnly:  true, MountPath: "/var/run/secrets/kubernetes.io/serviceaccount", }} // clear generated volume names for i := range pod.Spec.Volumes { if len(pod.Spec.Volumes[i].Name) > 0 { pod.Spec.Volumes[i].Name = "cleared" } } for i := range pod.Spec.Containers[0].VolumeMounts { if len(pod.Spec.Containers[0].VolumeMounts[i].Name) > 0 { pod.Spec.Containers[0].VolumeMounts[i].Name = "cleared" } } if !reflect.DeepEqual(expectedVolumes, pod.Spec.Volumes) { t.Errorf("unexpected volumes: %s", diff.ObjectReflectDiff(expectedVolumes, pod.Spec.Volumes)) } if !reflect.DeepEqual(expectedVolumeMounts, pod.Spec.Containers[0].VolumeMounts) { t.Errorf("unexpected volumes: %s", diff.ObjectReflectDiff(expectedVolumeMounts, pod.Spec.Containers[0].VolumeMounts)) } } </ins> func TestFetchesUncachedServiceAccount(t *testing.T) { ns := "myns"

What happened: We enabled the feature gate on to start switching from non-expiring to expiring ServiceAccount tokens as . It works well. We then wanted to completely get rid of the "legacy" non-expiring ServiceAccount token Secrets. To do that we configured with (so it wouldn't auto-create the Secrets), removed the existing Secrets and removed the reference from the respective ServiceAccount to the Secret. This works as expected: No non-expiring token Secrets are generated. However, when creating a Pod we now see: This happens because the rejects the Pod because it cannot find a ServiceAccount token Secret (because it doesn't exist anymore as intended). What you expected to happen: When using the feature the legacy ServiceAccount token Secrets aren't used anymore, so a Pod should not be rejected if the Secret doesn't exist. The Pod doesn't even use the Secret at all. How to reproduce it (as minimally and precisely as possible): Enable feature gate Start with Create a new ServiceAccount Deploy a dummy app using that ServiceAccount Observe no Pod being created with Describe the corresponding ReplicaSet to see the above Error event. Anything else we need to know?: Looking at the it seems that at least one legacy Secret must exist in order for the Pod to get admitted. This is enforced even when and we're relying on projected token volumes. Following the code path when is enabled reveals that the token is never going to be used later on as there's no SecretVolume to be mounted in . In other words:  is just fine when . During the transition period it's fine to have the legacy Secrets lying around but ultimately it's desirable to get rid of them because as far as I can tell they are still happily accepted by the API server. Environment: Kubernetes version: tested on but code seems the same in and Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): 4.15.0-1054-aws Install tools: /kind bug /sig auth
Fixed in
Thank you!


Question: ETCD_VERSION=${ETCD_VERSION:-"3.0.4"} # Define k8s version to use. <del> K8S_VERSION=${K8S_VERSION:-"1.1.1"} </del> <ins> K8S_VERSION=${K8S_VERSION:-"1.3.5"} DOCKER_DOWNLOAD_URL= "https://get.docker.com/builds/Linux/x86_64/docker-${DOCKER_VERSION}.tgz" </ins> FLANNEL_DOWNLOAD_URL= "https://github.com/coreos/flannel/releases/download/v${FLANNEL_VERSION}/flannel-${FLANNEL_VERSION}-linux-amd64.tar.gz"

Did a git clone and trying to build on centos. [root kubernetes]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root kubernetes]# make release +++ [0616 17:42:23] Verifying Prerequisites.... +++ [0616 17:42:57] Building Docker image kube-build:build-. +++ [0616 17:44:44] Running build command.... Usage of loopback devices is strongly discouraged for production use. Either use or use to suppress this warning. Go version: go version go1.6.2 linux/amd64 +++ [0616 17:45:01] Multiple platforms requested, but available 0G < threshold 11G, building platforms in serial +++ [0616 17:45:01] Building the toolchain targets: +++ [0616 17:45:04] Building go targets for linux/amd64: cmd/kube-dns cmd/kube-proxy cmd/kube-apiserver cmd/kube-controller-manager cmd/kubelet cmd/kubemark cmd/hyperkube federation/cmd/federation-apiserver federation/cmd/federation-controller-manager plugin/cmd/kube-scheduler go build /usr/local/go/pkg/tool/linuxamd64/compile: signal: killed !!! Error in 'go install "{goflags[ -ldflags "{nonstatics[ exited with status 1 Call stack: 1: kube::golang::buildbinariesforplatform(...) 2: kube::golang::buildbinaries(...) 3: hack/build- main(...) Exiting with status 1 !!! Error in '( kube::golang::setupenv; echo "Go version: (kube::golang::hostplatform); local goflags goldflags; eval "goflags=({KUBEGOLDFLAGS:-} do if [[ "${arg}" == "--usegobuild" ]]; then usegobuild=true; else if [[ "${arg}" == - ]]; then goflags+=("${arg}"); else targets+=("${arg}"); fi; fi; done; if [[ ${#targets[ -eq 0 ]]; then targets=("${KUBEALLTARGETS[ fi; local -a platforms=("{KUBEBUILDPLATFORMS[ if [[ ${#platforms[ -eq 0 ]]; then platforms=("${hostplatform}"); fi; local binaries;

binaries=({targets[ local parallel=false; if [[ ${#platforms[ -gt 1 ]]; then local gigs; gigs={gigs} -ge ${KUBEPARALLELBUILDMEMORY} ]]; then kube::log::status "Multiple platforms requested and available {KUBEPARALLELBUILDMEMORY}G, building platforms in parallel"; parallel=true; else kube::log::status "Multiple platforms requested, but available {KUBEPARALLELBUILDMEMORY}G, building platforms in serial"; parallel=false; fi; fi; kube::golang::buildkubetoolchain; if [[ "${parallel}" == "true" ]]; then kube::log::status "Building go targets for {targets[ local platform; for platform in "${platforms[ do ( kube::golang::setplatformenvs "{platform}: go build started"; kube::golang::buildbinariesforplatform {usegobuild:-}; kube::log::status "{platform////}.build" & done; local fails=0; for job in $(jobs -p); do wait ${job} let "fails+=1"; done; for platform in "${platforms[ do cat "/tmp//${platform////}.build"; done; exit ${fails}; else for platform in "${platforms[ do kube::log::status "Building go targets for {targets[ kube::golang::setplatformenvs "{platform} ${usegobuild:-}; done; fi )' exited with status 1 Call stack: 1: kube::golang::buildbinaries(.) 2: hack/build- main(.) Exiting with status 1 !!! Error in '" exited with status 1 Call stack: 1: kube::build::runbuildcommand(...) 2: main(...) Exiting with status 1 make:  [release] Error 1
Note that Centos downloads a bad "docker" binary that is an image with "404 Error". Centos seems abandonned by Kubernetes team... so I needed to make the entire operation with Ansible. And that is not simple at all :)
There was below info from your log: It seems like that the compiler was OOM killed due to memory pressure. AFAIK, consumes a lot of memory. I used to encounter when running it. We're going to close this issue.

Feel free to file a new issue if you fail to run at latest HEAD and you think there is a bug. Thanks!


Question: obj := &corev1.Pod{} newObj, err := f.Apply(obj, []byte(`{ <del> "apiVersion": "v1", </del> <ins> "apiVersion": "apps/v1", </ins> "kind": "Deployment", "metadata": { "name": "b",

What happened: When sending a request to the server side apply endpoint (), if the apiVersion provided in the object is different from the apiVersion in the url, it returns a 500 error, like: What you expected to happen: The request should fail if the apiVersion provided in the object is different from the apiVersion in the url, but it should fail before trying to merge, and be a 4xx error. Probably a bad request error. Anything else we need to know?: The check should probably be done somewhere near this . The issue, though, is that the patch is given as a byte array, and needs to be deserialized into an object to access the apiVersion field. How to reproduce it (as minimally and precisely as possible): With a kube-apiserver built from head, running with :
/kind bug /sig api-machinery /wg apply /priority important-longterm /assign


Question: RevisionHistoryAnnotation:      true, DesiredReplicasAnnotation:      true, MaxReplicasAnnotation:          true, <del> OverlapAnnotation:              true, </del> } // skipCopyAnnotation returns true if we should skip copying the annotation with the given annotation key

The Deployment overlap annotation () is no longer populated nor used by 1.6+ on the server side. However, in we maintained knowledge of the annotation for 1.6 clients in case they talk to pre-1.6 servers. That PR also adds code to clean up the annotation whenever it's found in 1.6+ clusters, so that pre-1.6 clients don't see it and make the wrong assumptions. The code handling this annotation should be removed when all the following conditions are met: We no longer support a 1.5.x client talking to the latest server version. We no longer support upgrading the server directly from 1.5.x to the latest version. The soonest milestone that meets both conditions according to the is v1.8. This issue should be moved to that milestone once it exists.
TODOs: [x] deployment controller: [x] deployment util: [x] deployment reaper: [x] deployment describer:
I can help on this
/assign
GitHub didn't allow me to assign the following users: wanghaoran1988. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks, feel free to send patches


Question: {Name: "jobs", Namespaced: true, Kind: "Job"}, }, } <ins> extensionsbeta3 := metav1.APIResourceList{GroupVersion: "extensions/v1beta3", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} extensionsbeta4 := metav1.APIResourceList{GroupVersion: "extensions/v1beta4", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} extensionsbeta5 := metav1.APIResourceList{GroupVersion: "extensions/v1beta5", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} extensionsbeta6 := metav1.APIResourceList{GroupVersion: "extensions/v1beta6", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} extensionsbeta7 := metav1.APIResourceList{GroupVersion: "extensions/v1beta7", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} extensionsbeta8 := metav1.APIResourceList{GroupVersion: "extensions/v1beta8", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} extensionsbeta9 := metav1.APIResourceList{GroupVersion: "extensions/v1beta9", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} extensionsbeta10 := metav1.APIResourceList{GroupVersion: "extensions/v1beta10", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} appsbeta1 := metav1.APIResourceList{GroupVersion: "apps/v1beta1", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} appsbeta2 := metav1.APIResourceList{GroupVersion: "apps/v1beta2", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} appsbeta3 := metav1.APIResourceList{GroupVersion: "apps/v1beta3", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} appsbeta4 := metav1.APIResourceList{GroupVersion: "apps/v1beta4", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} appsbeta5 := metav1.APIResourceList{GroupVersion: "apps/v1beta5", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} appsbeta6 := metav1.APIResourceList{GroupVersion: "apps/v1beta6", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} appsbeta7 := metav1.APIResourceList{GroupVersion: "apps/v1beta7", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} appsbeta8 := metav1.APIResourceList{GroupVersion: "apps/v1beta8", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} appsbeta9 := metav1.APIResourceList{GroupVersion: "apps/v1beta9", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} appsbeta10 := metav1.APIResourceList{GroupVersion: "apps/v1beta10", APIResources: []metav1.APIResource{{Name: "deployments", Namespaced: true, Kind: "Deployment"}}} </ins> tests := []struct { resourcesList *metav1.APIResourceList path          string

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened


Question: file := corev1.PersistentVolumeFilesystem goldClassName := "gold" now := time.Now() <del> deletionTimestamp := metav1.Time{Time: time.Now().UTC().AddDate(10, 0, 0)} </del> <ins> deletionTimestamp := metav1.Time{Time: time.Now().UTC().AddDate(-10, 0, 0)} </ins> testCases := []struct { name               string pvc                *corev1.PersistentVolumeClaim

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubectl describe pv xxx shows last<invalidif pv is terminating ! I found pvc and pod have the same issue What you expected to happen: Terminating (lasts 10s) How to reproduce it (as minimally and precisely as possible): 1 create a pv, pvc or pod is terminating 2 using kubectk describe xxx Anything else we need to know?: Environment: Kubernetes version (use ): 1.13.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig cli
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: // zone based on pod labels. }) <del> ginkgo.Describe("GCE [Slow] [Feature:NEG]", func() { </del> <ins> ginkgo.Describe("GCE [Slow] [Feature:NEG] [Flaky]", func() { </ins> var gceController *gce.IngressController // Platform specific setup

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Timeout while - trying to gave a 404 or 502 Anything else we need to know: 4 flakes in the past week: Triage: /sig network <!-- Please see the deflaking doc () for more guidance! --
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
Hi and any update on this ?
/assign
Minhan -- can you take a look?
/triage accepted
Looks like the LB did not stabilize before timeout. Still looking. We are considering disabling some of the tests here
This test is still flaking a bit -- recent failures from the past 2 days:
It looks like this test is still flaking quite frequently: Any ideas on how to move forward?
I will start disabling the tests.


Question: func (t *prefixTransformers) TransformToStorage(ctx context.Context, data []byte, dataCtx Context) ([]byte, error) { start := time.Now() transformer := t.transformers[0] <del> prefixedData := make([]byte, len(transformer.Prefix), len(data)+len(transformer.Prefix)) copy(prefixedData, transformer.Prefix) </del> result, err := transformer.Transformer.TransformToStorage(ctx, data, dataCtx) RecordTransformation("to_storage", string(transformer.Prefix), start, err) if err != nil { return nil, err } <ins> prefixedData := make([]byte, len(transformer.Prefix), len(result)+len(transformer.Prefix)) copy(prefixedData, transformer.Prefix) </ins> prefixedData = append(prefixedData, result...) return prefixedData, nil }

The code at: assumes that the transformation does not change the size of the data. Since we know it most cases that it does, we should try to account for that during the allocation.
/sig auth
/triage accepted
/assign
I think there is no problem with the memory allocation of , because needs to be executed afterwards, in order to avoid executing new memory allocation during append, so is no problem. IMO
After reading your code, I understand.
/assign


Question: return result } <ins> // getCompletionMode returns string representation of the completion mode. Used as a label value for metrics. func getCompletionMode(job *batch.Job) string { if isIndexedJob(job) { return string(batch.IndexedCompletion) } return string(batch.NonIndexedCompletion) } </ins> func trackingUncountedPods(job *batch.Job) bool { return feature.DefaultFeatureGate.Enabled(features.JobTrackingWithFinalizers) && hasJobTrackingAnnotation(job) }

Occasionally job completions are counted twice (or potentially multiple times?) in the metric. This happens for about 20% of jobs, but may depend on the Job specificity, which results in an overestimation of the number of finished jobs. The metric reflects the actual number of completed jobs. a kind cluster with prometheus monitoring stack (for example follow this: ). the number of completed jobs by inspecting the metric and delete a job multiple times in a loop Example job: Example repro script: . Check the number of completed jobs by inspecting the metric and the job Yes, in the kube-controller-logs we find the following log lines which are related and probably indicate repeated updated which results in double counting. <details</details<detailsReproducible with kind </details<details</details<details</details<details</details<details</details
/assign
FYI
/sig apps
/triage accepted


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/pkg/scheduler/internal/cache" <del> fakecache "k8s.io/kubernetes/pkg/scheduler/internal/cache/fake" </del> "k8s.io/kubernetes/pkg/scheduler/internal/queue" ) <del> func TestSkipPodUpdate(t *testing.T) { for _, test := range []struct { pod              *v1.Pod isAssumedPodFunc func(*v1.Pod) bool getPodFunc       func(*v1.Pod) *v1.Pod expected         bool name             string }{ { name: "Non-assumed pod", pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-0", }, }, isAssumedPodFunc: func(*v1.Pod) bool { return false }, getPodFunc: func(*v1.Pod) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-0", }, } }, expected: false, }, { name: "with changes on ResourceVersion, Spec.NodeName and/or Annotations", pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "pod-0", Annotations:     map[string]string{"a": "b"}, ResourceVersion: "0", }, Spec: v1.PodSpec{ NodeName: "node-0", }, }, isAssumedPodFunc: func(*v1.Pod) bool { return true }, getPodFunc: func(*v1.Pod) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "pod-0", Annotations:     map[string]string{"c": "d"}, ResourceVersion: "1", }, Spec: v1.PodSpec{ NodeName: "node-1", }, } }, expected: true, }, { name: "with ServerSideApply changes on Annotations", pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "pod-0", Annotations:     map[string]string{"a": "b"}, ResourceVersion: "0", ManagedFields: []metav1.ManagedFieldsEntry{ { Manager:    "some-actor", Operation:  metav1.ManagedFieldsOperationApply, APIVersion: "v1", FieldsType: "FieldsV1", FieldsV1: &metav1.FieldsV1{ Raw: []byte(` "f:metadata": { "f:annotations": { "f:a: {} } } `), }, }, }, }, Spec: v1.PodSpec{ NodeName: "node-0", }, }, isAssumedPodFunc: func(*v1.Pod) bool { return true }, getPodFunc: func(*v1.Pod) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "pod-0", Annotations:     map[string]string{"a": "c", "d": "e"}, ResourceVersion: "1", ManagedFields: []metav1.ManagedFieldsEntry{ { Manager:    "some-actor", Operation:  metav1.ManagedFieldsOperationApply, APIVersion: "v1", FieldsType: "FieldsV1", FieldsV1: &metav1.FieldsV1{ Raw: []byte(` "f:metadata": { "f:annotations": { "f:a: {} "f:d: {} } } `), }, }, { Manager:    "some-actor", Operation:  metav1.ManagedFieldsOperationApply, APIVersion: "v1", FieldsType: "FieldsV1", FieldsV1: &metav1.FieldsV1{ Raw: []byte(` "f:metadata": { "f:annotations": { "f:a: {} } } `), }, }, }, }, Spec: v1.PodSpec{ NodeName: "node-1", }, } }, expected: true, }, { name: "with changes on Labels", pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   "pod-0", Labels: map[string]string{"a": "b"}, }, }, isAssumedPodFunc: func(*v1.Pod) bool { return true }, getPodFunc: func(*v1.Pod) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   "pod-0", Labels: map[string]string{"c": "d"}, }, } }, expected: false, }, { name: "with changes on Finalizers", pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:       "pod-0", Finalizers: []string{"a", "b"}, }, }, isAssumedPodFunc: func(*v1.Pod) bool { return true }, getPodFunc: func(*v1.Pod) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:       "pod-0", Finalizers: []string{"c", "d"}, }, } }, expected: true, }, { name: "with changes on Conditions", pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-0", }, Status: v1.PodStatus{ Conditions: []v1.PodCondition{ {Type: "foo"}, }, }, }, isAssumedPodFunc: func(*v1.Pod) bool { return true }, getPodFunc: func(*v1.Pod) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-0", }, } }, expected: true, }, } { t.Run(test.name, func(t *testing.T) { c := &Scheduler{ SchedulerCache: &fakecache.Cache{ IsAssumedPodFunc: test.isAssumedPodFunc, GetPodFunc:       test.getPodFunc, }, } got := c.skipPodUpdate(test.pod) if got != test.expected { t.Errorf("skipPodUpdate() = %t, expected = %t", got, test.expected) } }) } } </del> func TestNodeAllocatableChanged(t *testing.T) { newQuantity := func(value int64) resource.Quantity { return *resource.NewQuantity(value, resource.BinarySI)

Currently the scheduler may add an assumed pod back to the queue if the update didn't include specific fields (see ). There doesn't seem to be a reason why we can't simply ignore all updates of assumed pods. Assumed pods are waiting to be bound, if binding succeeded then the pod is scheduled. If binding fails, the pod's latest version will be put back into the unschedulable queue anyways. Performance: we save on unnecessary scheduling attempts. This is a followup on the discussion we had in /sig scheduling /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : #### What would you like to be : /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What we basically need to do is: 1) remove function 2) in the places where we call , we should simply call
/assign


Question: return 0, framework.NewStatus(framework.Error, fmt.Sprintf("getting node %q from Snapshot: %v", nodeName, err)) } <del> meta := migration.PriorityMetadata(state) s, err := priorities.CalculateNodePreferAvoidPodsPriorityMap(pod, meta, nodeInfo) return s.Score, migration.ErrorToFrameworkStatus(err) </del> <ins> node := nodeInfo.Node() if node == nil { return 0, framework.NewStatus(framework.Error, "node not found") } controllerRef := metav1.GetControllerOf(pod) if controllerRef != nil { // Ignore pods that are owned by other controller than ReplicationController // or ReplicaSet. if controllerRef.Kind != "ReplicationController" && controllerRef.Kind != "ReplicaSet" { controllerRef = nil } } if controllerRef == nil { return framework.MaxNodeScore, nil } avoids, err := v1helper.GetAvoidPodsFromNodeAnnotations(node.Annotations) if err != nil { // If we cannot get annotation, assume it's schedulable there. return framework.MaxNodeScore, nil } for i := range avoids.PreferAvoidPods { avoid := &avoids.PreferAvoidPods[i] if avoid.PodSignature.PodController.Kind == controllerRef.Kind && avoid.PodSignature.PodController.UID == controllerRef.UID { return 0, nil } } return framework.MaxNodeScore, nil </ins> } // ScoreExtensions of the Score plugin.

/sig scheduling /priority important-soon See as an example. Part of This shouldn't need a PostFilter Plugin, it is not worth caching the result of metav1.GetControllerOf. /assign


Question: elif [[ "${KUBE_FASTBUILD:-}" == "true" ]]; then host_arch=$(kube::util::host_arch) <del> if [[ "${host_arch}" != "amd64" && "${host_arch}" != "arm64" ]]; then # on any platform other than amd64 and arm64, we just default to amd64 </del> <ins> if [[ "${host_arch}" != "amd64" && "${host_arch}" != "arm64" && "${host_arch}" != "ppc64le" ]]; then # on any platform other than amd64, arm64 and ppc64le, we just default to amd64 </ins> host_arch="amd64" fi KUBE_SERVER_PLATFORMS=("linux/${host_arch}")

On a ppc64le host, fast build was failing: # make KUBEFASTBUILD=true quick-release +++ [0908 15:56:36] Verifying Prerequisites.... +++ [0908 15:56:36] Building Docker image kube-build:build--5-v1.23.0-go1.17-buster.0 +++ [0908 15:56:44] Syncing sources to container +++ [0908 15:56:48] Running build command... +++ [0908 15:56:56] Building go targets for linux/ppc64le: touch: cannot touch 'output/bin/prerelease-lifecycle-gen': No such file or directory make[2]: [Makefile.generatedfiles:209: output/bin/prerelease-lifecycle-gen] Error 1 make[1]: [Makefile:552: generatedfiles] Error 2 make: [Makefile:512: cross] Error 1 !!! [0908 15:56:58] Call tree: !!! [0908 15:56:58] 1: kube::build::runbuildcommandex(...) !!! [0908 15:56:58] 2: kube::build::runbuildcommand(...) make: [Makefile:454: quick-release] Error 1 This error happened because the output/bin/ directory didn't exist at the moment the file output/bin/prerelease-lifecycle-gen was touched, so the path didn't exist. The output/bin symlink was not created by kube::golang::placebins() because kube::golang::setupplatforms() assumed KUBECLIENT_PLATFORMS=linux/amd64 despite being on a ppc64le host.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
looks like this is a missing feature and not a bug. /kind feature


Question: echo "Adding $minion to hosts file" echo "$ip $minion" >> /etc/hosts fi <del> echo "127.0.0.1 localhost" >> /etc/hosts # enables cmds like 'kubectl get pods' on master. </del> done <ins> echo "127.0.0.1 localhost" >> /etc/hosts # enables cmds like 'kubectl get pods' on master. </ins> # Update salt configuration mkdir -p /etc/salt/minion.d

It's a small problem. Now in "/etc/hosts" of master in vagrant cluster, there are duplicate "127.0.0.1 localhost". Though it doesn't make any error, I think it's better to remove duplicate content.


Question: readonly gcloud_supported_providers="gce gke" readonly master_logfiles="kube-apiserver.log kube-apiserver-audit.log kube-scheduler.log kube-controller-manager.log cloud-controller-manager.log etcd.log etcd-events.log glbc.log cluster-autoscaler.log kube-addon-manager.log konnectivity-server.log fluentd.log kubelet.cov" <del> readonly node_logfiles="kube-proxy.log containers/konnectivity-agent-*.log fluentd.log node-problem-detector.log kubelet.cov" </del> <ins> readonly node_logfiles="kube-proxy.log containers/konnectivity-agent-*.log fluentd.log node-problem-detector.log kubelet.cov kube-network-policies.log" </ins> readonly node_systemd_services="node-problem-detector" readonly hollow_node_logfiles="kubelet-hollow-node-*.log kubeproxy-hollow-node-*.log npd-hollow-node-*.log" readonly aws_logfiles="cloud-init-output.log"

,%20google-gce all I can't say , but at least a few weeks ,%20google-gce No response No response /sig network
/assign
/triage accepted


Question: MinReplicas:        o.Min, MaxReplicas:        o.Max, CPUPercent:         o.CPUPercent, <del> ScaleRefName:       name, </del> <ins> ScaleRefName:       refName, </ins> ScaleRefKind:       mapping.GroupVersionKind.Kind, ScaleRefAPIVersion: mapping.GroupVersionKind.GroupVersion().String(), }, nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: the hpa name is always info.Namenot --name What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig cli


Question: { name: "Remove node before its pods", operations: []operation{ <del> addNode(0), addNode(1), addPod(1), addPod(11), removeNode(1), updatePod(1), updatePod(11), removePod(1), removePod(11), </del> <ins> addNode(0), addNode(1), addPod(1), addPod(11), updateSnapshot(), removeNode(1), updateSnapshot(), updatePod(1), updatePod(11), removePod(1), removePod(11), </ins> }, expected: []*v1.Node{nodes[0]}, },

/sig scheduling What happened: reverted an old behavior to ensure Pod information is not lost when Nodes are recreated without its Pods getting recreated. A ghost Pod is kep in The problem is that we rely in this map for determining deleted Nodes when updating the snapshot in . The good thing is that the nodeinfo's are cloned into the snapshot, so we shouldn't have nil pointer accesses. But we might lose a few cycles trying to schedule on nodes that are deleted How to reproduce it (as minimally and precisely as possible): a node with a few pods. node without deleting pods. Result: The snapshot still has the node. Environment: Kubernetes version (use ): 1.18+
/assign
It actually produces the error: which would cause failed scheduling cycles (the map is not recreated, only the lists).


Question: less, err := isLess(iField, jField) if err != nil { <del> klog.Fatalf("Field %s in %T is an unsortable type: %s, err: %v", r.field, iObj, iField.Kind().String(), err) </del> <ins> klog.Exitf("Field %s in %T is an unsortable type: %s, err: %v", r.field, iObj, iField.Kind().String(), err) </ins> } return less }

I am getting a crash when trying to sort pod with kubectl on Not crashing and at least an error message <details<summaryCrash log</summary</details- Kubernetes version (use ): Cloud provider or hardware configuration: tried on kind latest and openshift 4.7 OS (e.g: ): debian sid Kernel (e.g. ): 5.11.x Install tools: NA Network plugin and version (if this is a network-related bug): NA Others: kubectl compiled from tip branch (as of this bug report) with GO version 1.16.5
/sig cli
Hey I'm not able to reproduce the crash, although I do get an error on running However, the following works: Could you try this and check if this crashes as well?
It definitively crashes it, did you test it in a repo where there is actual pod in there? that error would not show if there is nothing.
Yep Hmm, interesting. Will try to reproduce it again
Sorry, not sure how to remove the label but this is expected. When sorting if the type is not expected it will output errors using over which will logs to the FATAL, ERROR, WARNING, and INFO logs, including a stack trace of all running goroutines, then calls os.Exit(255). Arguments are handled in the manner of fmt.Printf; a newline is appended if missing.
Those labels are not set on the issue: <detailsIn response to : /remove-kind bug Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
You need to have multiple pods to reproduce this error. It won't happen with just one pod. For example: The error says the type can't be sorted, so that is the problem: As said, this error is generated intentionally.

Here is the code: The problem stems from the fact that  has no way to return an error, so it logs a fatal error which os.Exits. It seems like the problem is that the error message is too verbose. One solution could be to change the code to only print the error message and exit, instead of using which is what is outputting the stack trace. It probably doesn't make sense to dump a stack trace for something that is a user error.


Question: - apiGroups: - "" resources: <ins> - pods/finalizers verbs: - update - apiGroups: - "" resources: </ins> - persistentvolumeclaims verbs: - create

This test fails when OwnerReferencesPermissionEnforcement admission plugin is enabled: After the test creates a pod with an ephemeral volume, ephemeral volume controller in kube-controller-manager fails to create a PVC for it: The controller must have permissions to to be allowed to set with to Pods. The test passes. With : ENABLEADMISSIONPLUGINS="NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,OwnerReferencesPermissionEnforcement" ALLOWPRIVILEGED=true LOGLEVEL=4 hack/local-up- Kubernetes version (use ): 1.21.0 & today's master
/sig storage /triage accepted /priority important-soon


Question: } } <ins> func TestVersionCheck(t *testing.T) { f := NewTestFieldManager(t) obj := &corev1.Pod{} // patch has 'apiVersion: apps/v1' and live version is apps/v1 -> no errors _, err := f.Apply(obj, []byte(`{ "apiVersion": "apps/v1", "kind": "Deployment", }`), "fieldmanager_test", false) if err != nil { t.Fatalf("failed to apply object: %v", err) } // patch has 'apiVersion: apps/v2' but live version is apps/v1 -> error _, err = f.Apply(obj, []byte(`{ "apiVersion": "apps/v2", "kind": "Deployment", }`), "fieldmanager_test", false) if err == nil { t.Fatalf("expected an error from mismatched patch and live versions") } switch typ := err.(type) { default: t.Fatalf("expected error to be of type %T was %T", apierrors.StatusError{}, typ) case apierrors.APIStatus: if typ.Status().Code != http.StatusBadRequest { t.Fatalf("expected status code to be %d but was %d", http.StatusBadRequest, typ.Status().Code) } } } </ins> func TestApplyDoesNotStripLabels(t *testing.T) { f := NewTestFieldManager(t) obj := &corev1.Pod{} newObj, err := f.Apply(obj, []byte(`{ <del> "apiVersion": "v1", </del> <ins> "apiVersion": "apps/v1", </ins> "kind": "Pod", "metadata": { "labels": {

What happened: When sending a request to the server side apply endpoint (), if the apiVersion provided in the object is different from the apiVersion in the url, it returns a 500 error, like: What you expected to happen: The request should fail if the apiVersion provided in the object is different from the apiVersion in the url, but it should fail before trying to merge, and be a 4xx error. Probably a bad request error. Anything else we need to know?: The check should probably be done somewhere near this . The issue, though, is that the patch is given as a byte array, and needs to be deserialized into an object to access the apiVersion field. How to reproduce it (as minimally and precisely as possible): With a kube-apiserver built from head, running with :
/kind bug /sig api-machinery /wg apply /priority important-longterm /assign


Question: trigger scaling up and down the number of pods in your application. * New GUI (dashboard) allows you to get started quickly and enables the same functionality found in the CLI as a more approachable and discoverable way of  interacting with the system. Note: the GUI is eanbled by default for new cluster creation, however, it does not   interacting with the system. Note: the GUI is enabled by default in 1.2 clusters.        ## Other notable improvements

The says this about the new GUI: It does not what? Seems important!
Hmm... That's interesting. Will fix.


Question: {Type: "third"}, }, }, <ins> { name:          "empty_conditions", conditions:    []metav1.Condition{}, conditionType: "Foo", expected:      []metav1.Condition{}, }, </ins> } for _, test := range tests {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Attempting to remove a status condition from an already empty slice panics. An empty slice should be removed (i.e. removed idempotently) and without panicking. Originally found with , reproduced with ( at the time of writing) Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): Pop!OS 20.04 LTS Kernel (e.g. ): Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig api-machinery
cc
/assign
/triage accepted


Question: be evicted if the taint is removed before toleration time ends. release: v1.16 file: test/e2e/node/taints.go <ins> - testname: Node, resource lifecycle codename: '[sig-node] Node Lifecycle should run through the lifecycle of a node [Conformance]' description: Creating and Reading a Node MUST succeed with required name retrieved. Patching a Node MUST succeed with its new label found. Listing Nodes with a labelSelector MUST succeed with only a single node found. Updating a Node MUST succeed with its new label found. Deleting the Node MUST succeed and its deletion MUST be confirmed. release: v1.32 file: test/e2e/node/node_lifecycle.go </ins> - testname: PodTemplate, delete a collection codename: '[sig-node] PodTemplates should delete a collection of pod templates [Conformance]' description: A set of Pod Templates is created with a label selector which MUST

[x] two weeks soak end date : 2024-09-08 According to following APIsnoop query, there are two Node endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 2 endpoints. The e2e logs for this test are listed below. This query shows the following Node endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/sig node
/reopen Tracking the next steps to conformance
Reopened this issue. <detailsIn response to : /reopen Tracking the next steps to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen Tracking the final step
Reopened this issue. <detailsIn response to : /reopen Tracking the final step Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // +optional Tolerations []Toleration // HostAliases is an optional list of hosts and IPs that will be injected into the pod's hosts <del> // file if specified. This is only valid for non-hostNetwork pods. </del> <ins> // file if specified. </ins> // +optional HostAliases []HostAlias // If specified, indicates the pod's priority. "system-node-critical" and

there is a note above hostAliases that this feature does not work with hostNework=true, yet it seems it does: when testing with a hostNetwork pod + hostAliases, there are no errors in kubelet logs at --v=10 related to this, in fact i don't see any hostAlias entries at all in there. i could not find anything blocking this in the kubelet, in fact there is normal handling of hostNetwork: i could not find any validation performed on the hostAliases field of PodSpec if hostNetwork is true on the API level. i could not find any unit tests or e2e tests related to this. the godoc should contain valid information if the feature is not supposed to work, it must be patched to not have the entries in the hosts file (perhaps?) we should have tests that ensure this type of requirement is guaranteed these steps confirm that a static pod with hostNework (deployed by kubeadm) does make use of hostAliases. create a kubeadm single node cluster directly with or one with modify the to include:wait some time for the new static pod spec to resync find the PID of the kube-apiserverget the hosts file n/a latest master 1.30.0-alpha.0 not applicable <details</detailskubeadm containerd v1.6.14 kindnet add test: update godoc:
/sig node network cluster-lifecycle /kind documentation
as reported by the hostAliases sometimes are not mounted for hostNetwork static pods, which is concerning: related slack thread:
At minimum the comment is wrong I need to be updated and we need e2e tests :/
found this, support was a while back: i will PR the API docs.
We identified an edge case here with Kubernetes v1.29. If a cloud provider did not yet initialise the node's , the code for creating the file including the hostAliases does not get run. To be concrete, in: is empty in this case which leads to to be . I was able to trace it back to this change which leads to this behaviour for static pods: For confirmation: I reverted the above commit, rebuilt kubelet and it got back to the old behaviour.

there is no perfect solution, but reverting is wrong because opens another bug, we need to study this more carefully but in the meantime installers that know the nodes addresses beforehand should set the --node-ip flag or not use cloud-provider external
yeah, as i mentioned downstream, revering is out of the question. it fixes some long standing problems. we might have to document somewhere this limitation in a cloud provider env.
i think it might be possible for us to special case this so that the hostAlias management does not wait for the node IP in a CP env, under certain conditions. just a speculation.
PRs add test: update godoc:
/triage accepted looks like it is being worked on already


Question: return m } <ins> // NewLongQueueBroadcaster functions nearly identically to NewBroadcaster, // except that the incoming queue is the same size as the outgoing queues // (specified by queueLength). func NewLongQueueBroadcaster(queueLength int, fullChannelBehavior FullChannelBehavior) *Broadcaster { m := &Broadcaster{ watchers:            map[int64]*broadcasterWatcher{}, incoming:            make(chan Event, queueLength), stopped:             make(chan struct{}), watchQueueLength:    queueLength, fullChannelBehavior: fullChannelBehavior, } m.distributing.Add(1) go m.loop() return m } </ins> const internalRunFunctionMarker = "internal-do-function" // a function type we can shoehorn into the queue.

The standard event recorder, when recording an event, calls the underlying Broadcaster in a goroutine, so that even recording is non-blocking. Things like leader election emit events over the course of their lifecycle via this broadcaster. Consider the following scenario: GR1: spawn leader election ----GR2 GR2: Leader election: emit event "Lease Acquired" ----spawn GR3 () GR1: Shutdown leader election ----blocks until GR2 is done GR1: Shutdown broadcaster --() GR3: ----send on closed channel () Practically, we've observed this behavior popping up occaisionally in tests in controller-runtime:
/sig api-machinery /kind bug
<details<summaryrelevant race detector output/stack traces</summary</details
were you planning on sending a fix, or need our help to send one to fix it?
I can probably try to figure out a fix if you want, but the whole model for how event broadcasters currently work is inverse to how Go suggests you manage channels (only writers should ever close channels, not readers). I'll need to do a bit of playing around with things.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think this is fixed by . Please reopen if it's not the case.
/triage accepted
Would be possible to cherry-pick this fix in 1.20?


Question: r.imageClientV1alpha2 = runtimeapiV1alpha2.NewImageServiceClient(conn) } else { <del> return fmt.Errorf("unable to determine image API version: %w", err) </del> <ins> return fmt.Errorf("unable to determine image API version with %q, or API is not implemented: %w", endpoint, err) </ins> } return nil

When installing the latest component, got the error: Then the flag , got error: It is confusing to see these error msgs at the first glance, need to make it more clearly. Add the required field validation, and add more supportive error message. Download the latest v1.25.0 binary, and install it with systemd service or just manually start the binary, will reproduce these errors. <details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
kubelet 1.24+ is not working with the docker, unless using Are you using it? If not, try switching to other available container runtimes. Please re-open if you have further questions /close
Closing this issue. <detailsIn response to : kubelet 1.24+ is not working with the docker, unless using Are you using it? If not, try switching to other available container runtimes. Please re-open if you have further questions Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, I got to know the , so need to add runtime validation by the fix PR: PTAL thanks.