Question: r.setLastSyncResourceVersion(resourceVersion) resyncerrc := make(chan error, 1) <ins> cancelCh := make(chan struct{}) defer close(cancelCh) </ins> go func() { for { select { case <-resyncCh: case <-stopCh: return <ins> case <-cancelCh: return </ins> } glog.V(4).Infof("%s: forcing resync", r.name) if err := r.store.Resync(); err != nil {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): watch leak Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 0 Environment: Cloud provider or hardware configuration: vagrant on Mac OS X OS (e.g. from /etc/os-release): CoreOS 1164.1.0 Kernel (e.g. ): Install tools: Others: What happened: We ran controller which handles our own ThirdPartyResource. When we hit "watch of ... ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared" error, "forcing resync" log is duplicated. This log message comes from . It looks like this goroutine does not exit when we hit the above mentioned error, and continued to work, and running "resync" more frequently. What you expected to happen: The resync goroutine must be terminated when watch is closed (i.e., ListAndWatch function returned), instead of leaking it. How to reproduce it (as minimally and precisely as possible): I just run our controller for ~15 minutes. I'm not sure why "401: The event in requested index is outdated and cleared (the requested history has been cleared" happens, but from the issues I read, it should be harmless. Anything else do we need to know:
I think I see what is going on here.

If exits for any reason beyond an error from the resync loop no one will tell the resync loop to exit. Furthermore, there is no "local" way to do this as it would be unexpected for this function to signal the Done channel as other things outside of it may be relying on it. There are some useful functions around Context () that help to create a "local" copy of context that can be used in this situation.
Thank you for reply. Isn't is possible to create local channel in ListAndWatch, and defer close it, and then make resync goroutine "select" it too?
This issue was labelled as component/kubectl, but actually it is not limited to kubectl. All controllers suffer from this. Only people notice this if they enable V(4) logging.
The really annoying thing caused by this issue is that eventually resync is done in every few seconds due to the leaked goroutine. Even if you set 30 second resync period in controller, leaked goroutines call resync too, and you end up having resync call very very frequently. If you have many resources in cache, it really performs unnecessary sync loop. This is the last thing you want.
this looks like a real issue. Could be related to Could you help triage? looks like a proper fix.
I agree - this looks like a real issue. And indeed LGTM as a fix for it.


Question: ) const maxNumberOfPods int64 = 10 <del> const minPodCPURequest int64 = 500 </del> var localStorageVersion = utilversion.MustParseSemantic("v1.8.0-beta.0")

cc cc Seeing a failure on our downgrade test. Can someone please look at this? https://k8s- Example failure https://k8s- Seeing timeouts on the downgrade test
Since we're so late in the release process and this is the first time we're seeing this fail, going to mark as non-release-blocking for now.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/scheduling: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
It fails in more places than just the downgrade job, but when the downgrade job fails that test, that timeout appears to be the consistent error that it fails with
/assign I looked at the logs. The test creates a few pods to saturate the existing nodes and then creates an additional pod and ensures that this last pod stays pending. The problem we observe is that those saturating pods are often not scheduled, causing a test timeout. Those pods use all the resources on the nodes. So, any small changes to the amount of available node resources or even rounding errors could cause those saturating pods to not schedule. I will test this theory by cutting down the amount of resources that the saturating pods use, but I would like to check if the problem is fixed before submitting the PR. Where can I get the steps these cluster downgrade tests take?
The test setups are in the test-infra repo. E.g.
see my analysis in .


Question: <ins> /* Copyright 2014 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package common import ( "fmt" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/test/e2e/framework" imageutils "k8s.io/kubernetes/test/utils/image" . "github.com/onsi/ginkgo" ) var _ = Describe("[sig-storage] Projected combined", func() { f := framework.NewDefaultFramework("projected") // Test multiple projections /* Release : v1.9 Testname: Projected Volume, multiple projections Description: A Pod is created with a projected volume source for secrets, configMap and downwardAPI with pod name, cpu and memory limits and cpu and memory requests. Pod MUST be able to read the secrets, configMap values and the cpu and memory limits as well as cpu and memory requests from the mounted DownwardAPIVolumeFiles. */ framework.ConformanceIt("should project all components that make up the projection API [Projection][NodeConformance]", func() { var err error podName := "projected-volume-" + string(uuid.NewUUID()) secretName := "secret-projected-all-test-volume-" + string(uuid.NewUUID()) configMapName := "configmap-projected-all-test-volume-" + string(uuid.NewUUID()) configMap := &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{ Namespace: f.Namespace.Name, Name:      configMapName, }, Data: map[string]string{ "configmap-data": "configmap-value-1", }, } secret := &v1.Secret{ ObjectMeta: metav1.ObjectMeta{ Namespace: f.Namespace.Name, Name:      secretName, }, Data: map[string][]byte{ "secret-data": []byte("secret-value-1"), }, } By(fmt.Sprintf("Creating configMap with name %s", configMap.Name)) if configMap, err = f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Create(configMap); err != nil { framework.Failf("unable to create test configMap %s: %v", configMap.Name, err) } By(fmt.Sprintf("Creating secret with name %s", secret.Name)) if secret, err = f.ClientSet.CoreV1().Secrets(f.Namespace.Name).Create(secret); err != nil { framework.Failf("unable to create test secret %s: %v", secret.Name, err) } pod := projectedAllVolumeBasePod(podName, secretName, configMapName, nil, nil) pod.Spec.Containers = []v1.Container{ { Name:    "projected-all-volume-test", Image:   imageutils.GetE2EImage(imageutils.BusyBox), Command: []string{"sh", "-c", "cat /all/podname && cat /all/secret-data && cat /all/configmap-data"}, VolumeMounts: []v1.VolumeMount{ { Name:      "podinfo", MountPath: "/all", ReadOnly:  false, }, }, }, } f.TestContainerOutput("Check all projections for projected volume plugin", pod, 0, []string{ fmt.Sprintf("%s", podName), "secret-value-1", "configmap-value-1", }) }) }) func projectedAllVolumeBasePod(podName string, secretName string, configMapName string, labels, annotations map[string]string) *v1.Pod { pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:        podName, Labels:      labels, Annotations: annotations, }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: "podinfo", VolumeSource: v1.VolumeSource{ Projected: &v1.ProjectedVolumeSource{ Sources: []v1.VolumeProjection{ { DownwardAPI: &v1.DownwardAPIProjection{ Items: []v1.DownwardAPIVolumeFile{ { Path: "podname", FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "metadata.name", }, }, }, }, }, { Secret: &v1.SecretProjection{ LocalObjectReference: v1.LocalObjectReference{ Name: secretName, }, }, }, { ConfigMap: &v1.ConfigMapProjection{ LocalObjectReference: v1.LocalObjectReference{ Name: configMapName, }, }, }, }, }, }, }, }, RestartPolicy: v1.RestartPolicyNever, }, } return pod } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Running the non- conformance tests with produces unpredictable results, sometimes tests show up again with a on the end. This is due to the following tests having multiple tests with identical names: - - What you expected to happen: Tests should have unique names. How to reproduce it (as minimally and precisely as possible): run the conformance tests with a skip on and Anything else we need to know?: example here: https://k8s- /sig node /sig testing cc
er this is probably better /sig storage cc
/assign /assign
/area conformance
/milestone v1.13


Question: "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/watch" <ins> "k8s.io/apiserver/pkg/authentication/group" </ins> "k8s.io/apiserver/pkg/authentication/request/bearertoken" "k8s.io/apiserver/pkg/authentication/token/tokenfile" "k8s.io/apiserver/pkg/authentication/user"

While moving APF to beta (see comments on ), we discovered that the integration tests commonly present users which are not members of any groups. We had thought that the system enforced membership in either the authenticated or unauthenticated group. If this is supposed to be a system invariant (as we thought), we'd like to ask SIG Auth: Please fix the integration tests Please add e.g. logging so that if real, operational clusters hit this there's at least a log message and people will file bugs. /sig auth
Can I suggest also, if this is a real invariant, that it be documented in some place(s) likely to be read?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /priority important-longterm /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen
fixes the authenticator configured in one of the integration test API server startup helpers which resolved this issue for . Remaining tests that still need fixing that are encountering unexpected users look to be ones set up to specifically test auth paths:
open to fix remaining integration test users


Question: } defer backendConn.Close() <ins> // determine the http response code from the backend by reading from rawResponse+backendConn rawResponseCode, headerBytes, err := getResponseCode(io.MultiReader(bytes.NewReader(rawResponse), backendConn)) if err != nil { klog.V(6).Infof("Proxy connection error: %v", err) h.Responder.Error(w, req, err) return true } if len(headerBytes) > len(rawResponse) { // we read beyond the bytes stored in rawResponse, update rawResponse to the full set of bytes read from the backend rawResponse = headerBytes } </ins> // Once the connection is hijacked, the ErrorResponder will no longer work, so // hijacking should be the last step in the upgrade. requestHijacker, ok := w.(http.Hijacker)

(9.8, critical) With a specially crafted request, users that are authorized to establish a connection through the Kubernetes API server to a backend server can then send arbitrary requests over the same connection directly to that backend, authenticated with the Kubernetes API server’s TLS credentials used to establish the backend connection. Thanks to Darren Shepherd for reporting this problem. CVE-2018- is fixed in the following Kubernetes releases: Kubernetes API server Kubernetes v1.0.x-1.9.x Kubernetes v1.10.0-1.10.10 (fixed in ) Kubernetes v1.11.0-1.11.4 (fixed in ) Kubernetes v1.12.0-1.12.2 (fixed in ) Note: If you are using binaries or packages provided by a distributor (not the ones provided in the open source release artifacts), you should contact them to determine what versions resolve this CVE. Distributors may choose to provide support for older releases beyond the ones maintained by the open source project. Clusters = 1.6.x that run aggregated API servers (like the metrics server) that are directly accessible from the Kubernetes API server’s network. If there are aggregated API servers configured in a cluster, the following command will return the names of the associated APIService objects (if no names are listed, or the kube-apiserver is an older version that does not have the apiservices API, then the cluster has no aggregated API servers configured): Clusters = 1.0.x that grant pod exec/attach/portforward permissions to users that are not expected to have full access to kubelet APIs An API call to any aggregated API server endpoint can be escalated to perform any API request against that aggregated API server, as long as that aggregated API server is directly accessible from the Kubernetes API server’s network. allows all users (authenticated and unauthenticated) to perform discovery API calls that allow this escalation against any aggregated API servers configured in the cluster. A pod exec/attach/portforward API call can be escalated to perform any API request against the kubelet API on the node specified in the pod spec (e.g. listing all pods on the node, running arbitrary commands inside those pods, and obtaining the command output). Pod exec/attach/portforward permissions are included in the admin/edit RBAC roles intended for namespace-constrained users. This section lists possible mitigations to use prior to upgrading.

Note that many of the mitigations are likely to be disruptive, and upgrading to a fixed version is strongly recommended. suspend use of aggregated API servers (note that this will disrupt users of the APIs provided by the aggregated server) disable anonymous requests by passing to the kube-apiserver (note that this may disrupt load balancer or kubelet health checks of the kube-apiserver, and breaks setup flows) remove all anonymous access to all aggregated APIs (including discovery permissions granted by the ) suspend use of aggregated API servers (note that this will disrupt users of the APIs provided by the aggregated server) remove all access to all aggregated APIs (including discovery permissions granted by the ) from users that should not have full access to the aggregated APIs (note that this may disrupt users and controllers that make use of discovery information to map API types to URLs) Remove pod exec/attach/portforward permissions from users that should not have full access to the kubelet API There is no simple way to detect whether this vulnerability has been used. Because the unauthorized requests are made over an established connection, they do not appear in the Kubernetes API server audit logs or server log. The requests do appear in the kubelet or aggregated API server logs, but are indistinguishable from correctly authorized and proxied requests via the Kubernetes API server.
, why doesn't kubelet's webhook authorization help here? Shouldn't it reject subsequent requests on still open TCP connection from APIServer?
Because the kube-apiserver <-kubelet connection was established with the kube-apiserver's TLS credentials, which are broadly authorized against the kubelet API. The kubelet would authorize the kube-apiserver to make that request, and allow it.
Hmm, I misunderstood how webhook authorization works then, I thought that no requests are trusted in that mode and even for /proxy requests, initiator's credentials are passed by APIServer to kubelet, which then comes back to apiserver for SubjectAccessReview. So I thought scheme is like following: Initiator -APIServer (authn/authz) -kubelet -APIServer (SubjectAccessReview) But seems it is not the case :(
the credentials authorized by the kubelet in that case are the apiserver's
Thanks for the fix and the detailed summary! Just for clarification - this vulnerability can't be exploited by anonymous users from outside of the cluster, as long as we disable , right?

I'm asking because my users tend to open up K8S API endpoint to the Internet, occasionally but not always restricting accesses to the endpoint at L4 by the source IP.
Correct, escalations start with a single authorized request through kube-apiserver to the backend. Preventing that (either by disallowing anonymous requests with --anonymous-auth=false or by removing the default discovery permissions granted to anonymous users) mitigate the anonymous escalation. Be aware that can disrupt normal operations of a cluster, including load balancer and kubelet detection of kube-apiserver health (via anonymous requests to /healthz), and kubeadm join flows that depend on public access to a config map containing cluster information in the namespace.
Initiator -APIServer (authn/authz) -kubelet -APIServer (SubjectAccessReview) Interesting, are there any drawbacks of passing initiator's credentials all the way to kubelet? Would it allow to eliminate one more "superuser" in the form of APIServer?
Exposing initiator's credentials broadly to the cluster is a non-goal. Additionally, many API callers have non-forwardable credentials (TLS client certificates). The apiserver could forward information about the initiator to the kubelet, but that information would have to be trusted (e.g. initiator info in headers, verified by mTLS connection proving the apiserver provided the info). Interestingly, that is exactly the scenario exercised by kube-apiserver -aggregated API server connections, where the kube-apiserver acts as an authenticating proxy, and the aggregated API server trusts the identity assertions sent to it by the kube-apiserver. not really, as long as kube-apiserver is responsible for terminating and asserting authentication information to backends.
- I'm having a hard time understanding the full extent of this. If the API is in a private network (no public exposure), with RBAC enabled, but still around, would it still be vulnerable to this?
The first step in the escalation is a properly authorized request to the kube-apiserver that gets proxied to a backend server. A malicious request must be able to reach the kube-apiserver for the vulnerability to be exploited.
That's what I thought, thanks so much for taking the time to answer so quickly!
Thanks for the clarification !
Hi, I want to know why the issue isn't fixed in 1.9.x?

We have a lot of clusters running the 1.9.x k8s.
Pardon stupid question, but is Heapster considered aggregated API server same as metrics server?
The Kubernetes project supports the most recent three minor releases with bug and security fixes. The currently supported releases are 1.10.x, 1.11.x, and 1.12.x. If you are running builds provided by a distributor (not the binaries or packages provided by the Kubernetes project), contact their support to determine which of their versions are vulnerable. Some distributors may apply fixes to versions prior to the open-source-supported versions. No, Heapster is not an aggregated API server. Aggregated API servers register objects with non-nil fields, and requests to endpoints on the kube-apiserver are proxied to the appropriate aggregated server.
Are there any detail procedures to reproduce the issue? So we can verify that if we have fixed it on 1.9.x. Thanks a lot.
How to reproduce the issue, so we can check if the kubernetes version we're using is affected by this issue? An example script/command shows how gets all pods information on that node can be helpful.
Is it enough to just update the API Server to a fixed version, or should all the components (scheduler, kubelet, etc.) be updated to the fixed Kubernetes version to be secure?
, fix is in the apiserver codebase, so it should be enough to update just it
Does this affect ELB/ALB/Ingress controllers in any way? I mean if K8s API is hidden in a private network but load balancers for services (not for K8s API) are wide open, is it possible to use this vulnerability?
If the request cannot reach the API servers, the exploit cannot work. Even if the request can reach the API servers, but not have the permission to establish backend/proxy connection, it is also fine. But if a user in your cluster can use kubectl "freely", it means that they can connect to K8s API server, and the CVE may be applied here. Or, if the services that run on your cluster has a role that allows pod exec/attach/port-forward, it can proxy the request from the service to the API server.

Or if one of your services are API server extension, and your users can interact with it directly, then he/she can craft the exploiting request
Could someone give an example of an aggregated API endpoint that is usually enabled by default?
the metrics server is an aggregated API server, so is often a valid endpoint
What is the safest way to "suspend use of aggregated API servers" and to "remove all anonymous access to all aggregated APIs" () ? Is it sufficient to move aside the proxy-client-key-file mentioned here: ?
Don't laugh at this question. is Kubernetes version 1.7.11 affected? I would assume so. But if anyone can confirm this will help me out considerably. Thanks James Sorry I won't add another comment, I'll edit this one to thank you for your reply below, I mis-read the post on affected versions, I did not see 1.0.x-1.9.x, Sorry for wasting your time with the question.
I would say Yes, as Kubernetes v1.0.x-1.9.x 1.7.x falls within that range.
Sorry to bug you, but it would be a huge help to get an answer to my questions in .
I know the official docs on upgrading state to to from 1.7 to 1.8 then to 1.9 and then to 1.10, Would anyone know of a way to from 1.7.11 to 1.10.11. If not we will just figure out the steps and do the 3 separate upgrades. If something similar has been done before we'd like to hear about it as we have a few customers on 1.7.11 and all need to get upgraded. We will of course do our own trial and error in a lab and try and script much of it.
I don't understand why this leads to security issues. what is the "In default configurations"?
Without knowledge of your particular cluster and use, that is difficult to say. The safest way to suspend use of aggregated API servers might be to leave their objects registered, and scale down the deployments backing those objects, so that no aggregated server instances are available to receive requests.

Some potential disruptions that would cause: If portions of the cluster depend on an aggregated server (like HPA depending on metrics-server), this would break those dependencies aggregated servers that cannot be reached cause the garbage collector and namespace cleanup controllers to wait to perform some operations until the unavailable server can be reached again. This could prevent deletion of namespaces, and prevent foreground deletion of objects until the servers are made available again Removing anonymous access would either consist of disabling anonymous requests entirely with (see issue description for possible problems with that approach) or removing authorization permissions from anonymous users (specifically the user and group). If using , that includes removing the anonymous subjects from the and on restart. Keep in mind that preventing anonymous access still leaves the cluster vulnerable to escalation by authenticated users. Moving the would just prevent your kube-apiserver from starting Skip-version upgrading is not supported. A low-privilege request (like an anonymous call to a discovery API listing available resource types, like ) can be escalated to make any API call (including reading and writing API objects) to the aggregated server (like ). The default configuration refers to the discovery permissions granted to all users by .
on a 1.9.5 version i have it as --anonymous-auth=false and i deny all unauthenticated requests
In our Kubernetes cluster (Azure ACS, Kubernetes version# 1.8.4), we are using kubeconfig in order to establish proxy connection with kubernetes cluster. This is limited to only certain operations team members. It means in order to set connection with kubernetes api server, certificate from kubeconfig is used. Also, all the apis hosted on this cluster are exposed via azure api management service. In that case, is our kubernetes cluster still applicable to above vulnerability?
you mentioned "Aggregated API servers register APIService objects with non-nil spec.service fields" ... so is this a good query to see if someone has one of those in their environment?
This would work without jq: Added to the description
Gravitational has created a vulnerability test utility:
Under "affected configurations", it seems to suggest that if you run the given kubectl command and have no output, you do not have any exploitable endpoints for the aggregation api portion of this vulnerability. However, in "vulnerability impact" immediately below it states "In default configurations, all users (authenticated and unauthenticated) are allowed to perform discovery API calls that allow this escalation.".

Does this actually more precisely mean that the ability to discover vulnerable api calls that might exist is enough access to perform the exploit, if you happen to have exploitable endpoints? Asking because we appear to have no output from the kubectl command, but I can anonymously call "get --raw /apis" with . Unsure if this cluster is currently exploitable or not via this mechanism.
- clarified the "Affected configurations" and "Vulnerability impact" sections
thanks
Great thanks to !
there are two tests circulating for configurations that are vulnerable to the aggregated API exploits: the test you to the issue description, and which posted above. As mentioned in this issue: their results aren't always the same. In fact, we have clusters that are not affected according to your test but are affected according to . Could you clarify whether a negative from your test plus a positive from Gravitational's test indicates that we are not exposed?
the gravitational tests for ... however, if there are NO aggregated apis (as shown by 's command) then the cluster is still ok (hopefully can confirm this). You are still vulnerable to the other code path so you will have to review the RBAC permissions.
If there are no objects that route to a service (the check), that means no aggregated servers are configured in the cluster, and the anonymous escalation via aggregated discovery API requests is not possible. That said, if you are still running a vulnerable version and later add an aggregated API server, it would become exploitable.
Thanks very much for the clarity.
What about Kubernetes <1.5.x that no have "apiservices" daemon running/installed? And Could you please explain "user", which user do you mean. User that have shell access without root-access to kube-masters (what i think) and/or user from outside through an hosted application (ex. apache server on running pod-API-Server) and/or user that have general access to kube machines but without shell access. Thanks in advance
The APIService API was alpha in 1.6.x, and beta and enabled by default in 1.7.x+. kube-apiserver releases without an APIService API do not have the ability to proxy to aggregated API servers, so anonymous escalation via aggregated discovery API requests is not possible on those versions. Any actor that can send a network request to the kube-apiserver.

Escalation paths for this vulnerability begin with a network request to the kube-apiserver, which must first pass authorization, then be proxied to a backend (either to a kubelet in the case of pod exec/attach/portforward, or to an aggregated API server in the case of aggregated APIs).
The pod exec/attach/portforward escalation path applies back to 1.0.x
Fixed in for
Updated the description with a link to the document and recording of the post-mortem for this incident. Links to action items are in the document and reference this issue.
/label official-cve-feed (Related to )


Question: called  bool headers map[string][]string path    string <ins> host    string </ins> } func (d *targetHTTPHandler) Reset() { d.path = "" d.called = false d.headers = nil <ins> d.host = "" </ins> } func (d *targetHTTPHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { d.path = r.URL.Path d.called = true d.headers = r.Header <ins> d.host = r.Host </ins> w.WriteHeader(http.StatusOK) }

What happened: When apiserver is using apiservice and service EXTERNAL-IP to access the external http service(a sample metric service),the "http request header Host" is still the apiserver's Host, causing the external http proxy services are failed to forward the request normally. The requested link is like this client -apiserver -external http proxy service -real http service(metric service) What you expected to happen: Many http proxy servers handle request forwarding through the http request header Host, such as nginxif the aggregator accesses an external http service, the original request header HOST should be request.URL.Host How to reproduce it (as minimally and precisely as possible): apiservice and ExternalName service a http proxy server() and print the "HTTP HEADER HOST" to nginx accesslog to change test-ext- DNS IP to the proxy server ip https://127.0.0.1:6443/apis/power.metrics.extender/v1alpha1/nodes nginx access log,the "HTTP HEADER HOST" is '127.0.0.1:6443' but not test-ext- The following is the configuration NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.254.0.1 <none443/TCP 2d power-metrics ExternalName <nonetest-ext- <none2d Name: Namespace: Labels: <noneAnnotations: <noneAPI Version: Kind: APIService Metadata: Creation Timestamp: 2018-11-30T07:54:23Z Resource Version: Self Link: UID: -f475-11e8-a76f- Spec: Group: power.metrics.extender Group Priority Minimum: 1000 Insecure Skip TLS Verify: true Service: Name: power-metrics Namespace: default Version: v1alpha1 Version Priority: 15 Status: Conditions: Last Transition Time: 2018-12-03T05:51:12Z Message: all checks passed Reason: Passed Status: True Type: Available Events: <noneAnything else we need to know?

: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-30T07:43:21Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.0-alpha.0.2304+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-30T16:20:19Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" Kernel (e.g. ): Linux 4.9.87-linuxkit-aufs SMP Wed Mar 14 15:12:16 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery
/assign
so test-ext- is the proxy server, and from its access log, you see the req header is "127.0.0.1:6443", which is the apiserver's host. It seems that is working as expected. i.e, when apiserver sends the request to the proxy server, the req header should be apiserver's host. it cannot be the proxy server's host. Am I missing something here?

Thanks for your replyApiserver uses "test-ext-" to access proxy serverthe http req header's Host should be "test-ext-" but is "127.0.0.1:6443"this is not conform to IETF specification()
Yes, i see your point. I'll take a closer look at the code today or tomorrow.
Do you have any progress?
yes, i think the code is here: newReq.Header = utilnet.CloneHeader(req.Header) so the header is simply a clone of req.Header, though newReq.URL = location, and location.Host = rloc.Host, which has the correct resolved host for your case () Could you make a quick change to newReq.Header, and do a local build to check if it fixes the issue you encountered?
I think the problem code is here: copied Old 'req.Host' to New 'req.Host'then i add code:' ',Under line 144()it works for me. In additionI think the final request went out and passed this code this code Priority use 'req.Host' but not 'req.URL.Host'so apiserver needs to change req.Host externally before accessing external http services
I agree. Would you mind to make a PR?
ThanksI'll do it


Question: updatedDefaultClass1.GlobalDefault = false tests := []struct { <del> name                  string classesBefore         []*scheduling.PriorityClass classesAfter          []*scheduling.PriorityClass attributes            admission.Attributes expectedDefaultBefore int32 expectedDefaultAfter  int32 </del> <ins> name                      string classesBefore             []*scheduling.PriorityClass classesAfter              []*scheduling.PriorityClass attributes                admission.Attributes expectedDefaultBefore     int32 expectedDefaultNameBefore string expectedDefaultAfter      int32 expectedDefaultNameAfter  string </ins> }{ { <del> name:                  "simple resolution with a default class", classesBefore:         []*scheduling.PriorityClass{defaultClass1}, classesAfter:          []*scheduling.PriorityClass{defaultClass1}, attributes:            nil, expectedDefaultBefore: defaultClass1.Value, expectedDefaultAfter:  defaultClass1.Value, }, { name:                  "add a default class", classesBefore:         []*scheduling.PriorityClass{nondefaultClass1}, classesAfter:          []*scheduling.PriorityClass{nondefaultClass1, defaultClass1}, attributes:            admission.NewAttributesRecord(defaultClass1, nil, pcKind, "", defaultClass1.Name, pcResource, "", admission.Create, false, nil), expectedDefaultBefore: scheduling.DefaultPriorityWhenNoDefaultClassExists, expectedDefaultAfter:  defaultClass1.Value, }, { name:                  "multiple default classes resolves to the minimum value among them", classesBefore:         []*scheduling.PriorityClass{defaultClass1, defaultClass2}, classesAfter:          []*scheduling.PriorityClass{defaultClass2}, attributes:            admission.NewAttributesRecord(nil, nil, pcKind, "", defaultClass1.Name, pcResource, "", admission.Delete, false, nil), expectedDefaultBefore: defaultClass1.Value, expectedDefaultAfter:  defaultClass2.Value, }, { name:                  "delete default priority class", classesBefore:         []*scheduling.PriorityClass{defaultClass1}, classesAfter:          []*scheduling.PriorityClass{}, attributes:            admission.NewAttributesRecord(nil, nil, pcKind, "", defaultClass1.Name, pcResource, "", admission.Delete, false, nil), expectedDefaultBefore: defaultClass1.Value, expectedDefaultAfter:  scheduling.DefaultPriorityWhenNoDefaultClassExists, }, { name:                  "update default class and remove its global default", classesBefore:         []*scheduling.PriorityClass{defaultClass1}, classesAfter:          []*scheduling.PriorityClass{&updatedDefaultClass1}, attributes:            admission.NewAttributesRecord(&updatedDefaultClass1, defaultClass1, pcKind, "", defaultClass1.Name, pcResource, "", admission.Update, false, nil), expectedDefaultBefore: defaultClass1.Value, expectedDefaultAfter:  scheduling.DefaultPriorityWhenNoDefaultClassExists, </del> <ins> name:                      "simple resolution with a default class", classesBefore:             []*scheduling.PriorityClass{defaultClass1}, classesAfter:              []*scheduling.PriorityClass{defaultClass1}, attributes:                nil, expectedDefaultBefore:     defaultClass1.Value, expectedDefaultNameBefore: defaultClass1.Name, expectedDefaultAfter:      defaultClass1.Value, expectedDefaultNameAfter:  defaultClass1.Name, }, { name:                      "add a default class", classesBefore:             []*scheduling.PriorityClass{nondefaultClass1}, classesAfter:              []*scheduling.PriorityClass{nondefaultClass1, defaultClass1}, attributes:                admission.NewAttributesRecord(defaultClass1, nil, pcKind, "", defaultClass1.Name, pcResource, "", admission.Create, false, nil), expectedDefaultBefore:     scheduling.DefaultPriorityWhenNoDefaultClassExists, expectedDefaultNameBefore: "", expectedDefaultAfter:      defaultClass1.Value, expectedDefaultNameAfter:  defaultClass1.Name, }, { name:                      "multiple default classes resolves to the minimum value among them", classesBefore:             []*scheduling.PriorityClass{defaultClass1, defaultClass2}, classesAfter:              []*scheduling.PriorityClass{defaultClass2}, attributes:                admission.NewAttributesRecord(nil, nil, pcKind, "", defaultClass1.Name, pcResource, "", admission.Delete, false, nil), expectedDefaultBefore:     defaultClass1.Value, expectedDefaultNameBefore: defaultClass1.Name, expectedDefaultAfter:      defaultClass2.Value, expectedDefaultNameAfter:  defaultClass2.Name, }, { name:                      "delete default priority class", classesBefore:             []*scheduling.PriorityClass{defaultClass1}, classesAfter:              []*scheduling.PriorityClass{}, attributes:                admission.NewAttributesRecord(nil, nil, pcKind, "", defaultClass1.Name, pcResource, "", admission.Delete, false, nil), expectedDefaultBefore:     defaultClass1.Value, expectedDefaultNameBefore: defaultClass1.Name, expectedDefaultAfter:      scheduling.DefaultPriorityWhenNoDefaultClassExists, expectedDefaultNameAfter:  "", }, { name:                      "update default class and remove its global default", classesBefore:             []*scheduling.PriorityClass{defaultClass1}, classesAfter:              []*scheduling.PriorityClass{&updatedDefaultClass1}, attributes:                admission.NewAttributesRecord(&updatedDefaultClass1, defaultClass1, pcKind, "", defaultClass1.Name, pcResource, "", admission.Update, false, nil), expectedDefaultBefore:     defaultClass1.Value, expectedDefaultNameBefore: defaultClass1.Name, expectedDefaultAfter:      scheduling.DefaultPriorityWhenNoDefaultClassExists, expectedDefaultNameAfter:  "", </ins> }, }

Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup /sig scheduling Description: If Pod did not set , admission controller will set to "default value". If the default value are from default priority class, it's better to also set of pod.
can I take this? /assign
, thanks :) I created a PR for functionality, but that'll be great if you can help add some e2e test for it. I can help to shepherd :)
thanks let me have a try :)


Question: prometheus.NewGoCollector(), prometheus.NewProcessCollector(os.Getpid(), ""), ) <del> mux.Handle(prometheusEndpoint, promhttp.HandlerFor(r, promhttp.HandlerOpts{})) </del> <ins> mux.Handle(prometheusEndpoint, promhttp.HandlerFor(r, promhttp.HandlerOpts{ErrorHandling: promhttp.ContinueOnError})) </ins> } func staticHandlerNoAuth(w http.ResponseWriter, r *http.Request) {

Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug report Kubernetes version (use ): Environment: Cloud provider or hardware configuration: vsphere OS (e.g. from /etc/os-release):Kernel (e.g. ): 4.9.24-coreos Install tools: Others: What happened: After bringing a cluster up, there are no container metrics in the kubelet at /metrics What you expected to happen: I expect to see the metrics, but there are none How to reproduce it (as minimally and precisely as possible): Anything else we need to know: If I try and access the metrics in cAdvisor directly on port 4194 instead of on port then I see the following type of error for all of the metrics:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig node
This appears to be a v1.7 issue. If I run 1.6.6 I get these metrics. Seems like something has happened in 1.7. I'm running the 1.7 beta because I need nodePort for cni, so now I'm stuck.
Can you triage the issue since?
cc/
Can you help verifying is this is a new regression introduced in 1.7 cadvisor through node-e2e? Thanks!
I've tried this on both cos and cvm in 1.7 cluster, and no container metrics show up. However, I created a 1.6.4 GKE cluster, and container metrics are there. This is a regression in 1.7. Will figure out what is wrong here.
Here is a corresponding cadvisor issue
Even the user still can get the container metrics / stats from kubelet /summary api (recommended way), but since this is a regression introduced in 1.7 release, and we aren't dropping prometheus support, let's fix this for 1.7. Thanks!
Fixed in Will close this issue after building new cadvisor release and cherrypick into 1.7
Thanks for the quick find everyone! I'll be happy to give this a go on the next release!

I'm still facing this same issue. My kubernetes cluster is deployed with kubeadm on Centos 7. I run a k8S v1.8.1 cluster. When I try to fetch containers metrics this way: http://localhost:8001/api/v1/proxy/nodes/node1:0/metrics http://localhost:8001/api/v1/proxy/nodes/node1:/metrics http://localhost:8001/api/v1/proxy/nodes/node1:/metrics I find no metrics. kubelet is started with . If I start kubelet with . metrics are available through .


Question: import ( "fmt" <del> "k8s.io/apimachinery/pkg/api/resource" </del> ) <ins> type UsageInfo struct { Bytes  int64 Inodes int64 } </ins> // Info unsupported returns 0 values for available and capacity and an error. func Info(path string) (int64, int64, int64, int64, int64, int64, error) { return 0, 0, 0, 0, 0, 0, fmt.Errorf("fsinfo not supported for this build") } // DiskUsage gets disk usage of specified path. <del> func DiskUsage(path string) (*resource.Quantity, error) { return nil, fmt.Errorf("du not supported for this build") } // Find will always return zero since is on unsupported platform. func Find(path string) (int64, error) { return 0, fmt.Errorf("find not supported for this build") </del> <ins> func DiskUsage(path string) (UsageInfo, error) { var usage UsageInfo return usage, fmt.Errorf("directory disk usage not supported for this build.") </ins> }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: in calculates number of inodes wrong when there are hard links. What you expected to happen: inodes with hardlinks should only be counted once. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Issue is still not solved. There is an open PR waiting for review. /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
still waiting for to be merged

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to run pods on Alpine Linux with busybox find. What you expected to happen: I expected the call to exit with success, not failure. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing GNU findutils, so busybox implementation of is used. Anything else we need to know?: failed because the argument is not implemented in busybox . The option is not specified in so it can not be expected to be supported in all implementations. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related bug Others: I think it would be better to use instead of Related to
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This is solved by which is waiting for a review. /remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Still waiting for the PR to be merged. /remove-lifecycle rotten
/remove-lifecycle rotten

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running test suite on a cluster with workers on Alpine Linux, which uses busybox instead of GNU coreutils fails. The log shows that it fails to execute the command. What you expected to happen: I expected the call to exit with success. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing . Anything else we need to know?: It failed due to the option which is not implemented in busybox. The option is not specified in so I don't think it is reasonable to expect it to implemented in busybox. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related Others:
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale fixes this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten
Still waiting for PR to be merged


Question: return m.configuration.Load().([]webhook.WebhookAccessor) } <ins> // HasSynced returns true when the manager is synced with existing webhookconfig // objects at startup-- which means the informer is synced and either has no items // or updateConfiguration() has completed. </ins> func (m *mutatingWebhookConfigurationManager) HasSynced() bool { <del> return m.hasSynced() </del> <ins> if !m.hasSynced() { return false } if m.initialConfigurationSynced.Load().(bool) { // the informer has synced and configuration has been updated return true } if configurations, err := m.lister.List(labels.Everything()); err == nil && len(configurations) == 0 { // the empty list we initially stored is valid to use. // Setting initialConfigurationSynced to true, so subsequent checks // would be able to take the fast path on the atomic boolean in a // cluster without any admission webhooks configured. m.initialConfigurationSynced.Store(true) // the informer has synced and we don't have any items return true } return false </ins> } func (m *mutatingWebhookConfigurationManager) updateConfiguration() {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: Testgrid link: Example pass: Example fail: Reason for failure: Unknown Anything else we need to know: Looks like it's an apiserver test, but I'm not too sure. links to appreciated: <!-- Please see the deflaking doc () for more guidance! --/kind flake
Looks like is flaky too. Example fail:
/sig api-machinery
Thanks for the report. Please report individual failing tests/packages so the responsible sigs can be assigned. The cm/devicemanager issue is tracked in
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I don't see individual issues yet, but I do see open PRs targeting this issue /remove-lifecycle rotten
closing in favor of specific issues: /close
Closing this issue. <detailsIn response to : closing in favor of specific issues: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package apimachinery import ( "context" "fmt" "testing" "time" corev1 "k8s.io/api/core/v1" apiequality "k8s.io/apimachinery/pkg/api/equality" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/watch" "k8s.io/client-go/kubernetes" kubeapiservertesting "k8s.io/kubernetes/cmd/kube-apiserver/app/testing" "k8s.io/kubernetes/test/integration/framework" ) func TestWatchNamespaceEvent(t *testing.T) { timeout := 30 * time.Second server := kubeapiservertesting.StartTestServerOrDie(t, nil, nil, framework.SharedEtcd()) defer server.TearDownFn() clientSet, err := kubernetes.NewForConfig(server.ClientConfig) if err != nil { t.Fatal(err) } newWatcher := func(ctx context.Context, ns *corev1.Namespace, resourceVersion string) (watch.Interface, error) { return clientSet.CoreV1().Namespaces().Watch(ctx, metav1.ListOptions{ FieldSelector:   fields.OneTermEqualSelector("metadata.name", ns.Name).String(), ResourceVersion: resourceVersion, }) } newLegacyNSWatcher := func(ctx context.Context, ns *corev1.Namespace, resourceVersion string) (watch.Interface, error) { if resourceVersion == "" { return clientSet.CoreV1().RESTClient().Get(). RequestURI(fmt.Sprintf("/api/v1/watch/namespaces/%s", ns.Name)).Watch(ctx) } else { return clientSet.CoreV1().RESTClient().Get(). RequestURI(fmt.Sprintf("/api/v1/watch/namespaces/%s?resourceVersion=%s", ns.Name, resourceVersion)).Watch(ctx) } } newTestNamespace := func(name string) *corev1.Namespace { return &corev1.Namespace{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, } } tt := []struct { name            string namespace       *corev1.Namespace resourceVersion string getWatcher      func(ctx context.Context, ns *corev1.Namespace, resourceVersion string) (watch.Interface, error) }{ { name:       "watch namespace", namespace:  newTestNamespace("watch-namespace"), getWatcher: newWatcher, }, { name:            "watch namespace with resource version", namespace:       newTestNamespace("watch-ns-with-rv"), getWatcher:      newWatcher, resourceVersion: "0", }, { name:       "legacy watch namespace api", namespace:  newTestNamespace("legacy-watch-ns"), getWatcher: newLegacyNSWatcher, }, { name:            "legacy watch namespace api with resource version", namespace:       newTestNamespace("legacy-watch-ns-with-rv"), getWatcher:      newLegacyNSWatcher, resourceVersion: "0", }, } t.Run("group", func(t *testing.T) { for _, tc := range tt { tc := tc // we need to copy it for parallel runs t.Run(tc.name, func(t *testing.T) { t.Parallel() startTest := time.Now() ctx, cancel := context.WithTimeout(context.Background(), timeout) defer cancel() ns, err := clientSet.CoreV1().Namespaces().Create(ctx, tc.namespace, metav1.CreateOptions{}) if err != nil { t.Fatalf("Failed to create testing namespace %s: %v", tc.namespace.Name, err) } else { defer framework.DeleteNamespaceOrDie(clientSet, ns, t) } watcher, err := tc.getWatcher(ctx, ns, tc.resourceVersion) if err != nil { t.Fatalf("Failed to create watcher: %v", err) } defer watcher.Stop() t.Logf("start watching namespace %s event", tc.namespace.Name) generateAndWatchNamespaceEvent(ctx, t, clientSet, ns, watcher) t.Logf("Watch duration: %v", time.Since(startTest)) }) } }) } func generateAndWatchNamespaceEvent(ctx context.Context, t *testing.T, clientSet *kubernetes.Clientset, ns *corev1.Namespace, watcher watch.Interface) { timeout := 10 * time.Second nsName := ns.Name t.Logf("Expectiong to watch an add event") _, ok := waitForEvent(watcher, watch.Added, ns, timeout) if !ok { t.Fatalf("Failed to watch add event") } if ns.Annotations == nil { ns.Annotations = map[string]string{"e2e-watch-namespace": "update-v1"} } else { ns.Annotations["e2e-watch-namespace"] = "update-v1" } ns, err := clientSet.CoreV1().Namespaces().Update(ctx, ns, metav1.UpdateOptions{}) if err != nil { t.Fatalf("Failed to update namespace %s: %v", nsName, err) } t.Logf("Expectiong to watch an update event") _, ok = waitForEvent(watcher, watch.Modified, ns, timeout) if !ok { t.Fatalf("Failed to watch update event") } ns.Annotations["e2e-watch-namespace"] = "update-v2" ns, err = clientSet.CoreV1().Namespaces().Update(ctx, ns, metav1.UpdateOptions{}) if err != nil { t.Fatalf("failed to delete namespace %s: %v", nsName, err) } t.Logf("Expectiong to watch an update event") _, ok = waitForEvent(watcher, watch.Modified, ns, timeout) if !ok { t.Fatalf("Failed to watch update event") } } func waitForEvent(w watch.Interface, expectType watch.EventType, expectObject runtime.Object, duration time.Duration) (watch.Event, bool) { stopTimer := time.NewTimer(duration) defer stopTimer.Stop() for { select { case actual, ok := <-w.ResultChan(): if !ok { return watch.Event{}, false } if expectType == actual.Type && (expectObject == nil || apiequality.Semantic.DeepEqual(expectObject, actual.Object)) { return actual, true } case <-stopTimer.C: expected := watch.Event{ Type:   expectType, Object: expectObject, } return expected, false } } } </ins>

We upgraded our test cluster to (GKE ) and noticed that some of our tests began to time out when watching for events that never arrived. Upon investigation, it appears that there is a regression in the deprecated path-based watch API for at least e.g. . We have not been able to reproduce this with any other resource type. When creating a watch for a namespace (see ) we see the initial event, but then we do not see subsequent events if we modify the namespace e.g. add a label or annotation. I would expect to see events when the namespace is altered. E.g. adding a label should produce a event with the contents. a new test namespace e.g. a proxy to the master the watch path of the namespace that has been created e.g. another shell, modify the namespace to add a label to it. that there is no modified event in the watch. It is understood that the based paths are and have been for some time deprecated, we are working on moving to use the watch query param instead. <detailsWe are using Google Kubernetes Engine, please see version above. </detailsNo response No response No response No response
/sig api-machinery
The addWatcher add watch to allWatchers with with the value namespace not empty. But the function dispatch event with an empty namespace.
This issue is limited to watchers of the core namespace resource using the deprecated endpoint and watching a single namespace. This was introduced in 1.27 in (which was merged with test coverage in 1.27 in )
Hi, just to confirm this was working fine (and still is on our production clusters) on 1.28.7 it's only since 1.28.8 that we can see this bug, so slightly confused by your comment (and issue rename) to say 1.27+?
I reproduced the break at HEAD of release-1.27, and assumed the break came in with , but I was wrong.

release-1.26 HEAD: works v1.27.0: works release-1.27 HEAD: broken v1.28.0: works v1.28.7: works per your feedback v1.28.8: breaks per your feedback release-1.28 HEAD: broken That means that watch cache changes cherry picked back to 1.27 and 1.28 regressed something, and I don't know exactly what the breaking change was. The only thing that looks at all relevant between 1.28.7 and 1.28.8 is , but I don't see how that would only impact the single-item watch cc
Did you send the request with a resource version, or any other parameter? If the parameter is not empty, will work, or the old watcher will work. This makes a big difference.
This commit add the feature gate .
No, I just sent and that broke somewhere between 1.27.0 and
If you are asking me, I did not send any additional query parameter, no. On our cluster: works as expected, we see events when we add labels etc. On our cluster: does not work, we do not see events.
Ok, figured it out... 27.0 which is what made the watch cache not handle correctly (merged in with test coverage) also which routed watch requests without a resourceVersion parameter directly to etcd, bypassing watch cache 27.13 / 1.28.9 / 1.29.4 / 1.30.0 moved the behavior from behind an off-by-default gate because it was a massive performance regression. This had the side effect of making watch requests without a resourceVersion parameter get handled by the watch cache, and get exposed to this issue.

(GKE accelerated that specific fix internally to resolve that performance regression, so it showed up in a GKE 1.28.8-gke.X build, which is what confused me when looking for the delta between v1.28.7 and v1.28.8) in [1.27.0-1.12.12], [1.28.0-1.28.8], [1.29-1.29.3]: routes to etcd and works (because of ) routes to etcd and works (because of ) routes to watch cache and fails (where X is not 0) routes to watch cache and failsin 1.27.13+, 1.28.9+, 1.29.4+, 1.30.0+: routes to watch cache and fails routes to watch cache and fails routes to watch cache and fails (where X is not 0) routes to watch cache and fails
The workaround for anyone encountering this issue is to use the non-deprecated watch endpoint (and is a generally good change to make anyway) The Kubernetes fix for this I would recommend to restore pre-1.27 behavior is in my comment in Once that is done and test coverage , I'd be happy to see that backported to 1.27+
/cc
GitHub didn't allow me to assign the following users: tinatingyu. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: yt2985. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
just to verify, are you planning to take to completion?

if so, it would be ideal to complete that so the 1.27 backport can happen before 1.27 reaches EOL in June
I saw I was assigned with this issue. Taking a look...
sorry, was tagging you in since you helped bring in and might have context, but realized I should check with first since they have a fix in progress
Yew, working on it and writing test case.
cc
is this issue resolved already?
Not yet, is the proposed fix, under review now


Question: }{ { mode: kubeproxyconfig.ProxyMode("non-existing"), <del> msg:  "or blank (blank means the best-available proxy (currently iptables)", </del> <ins> msg:  "or blank (blank means the", </ins> }, } for _, errorCase := range errorCases { if errs := validateProxyMode(errorCase.mode, newPath.Child("ProxyMode")); len(errs) == 0 { <del> t.Errorf("expected failure for %s", errorCase.msg) </del> <ins> t.Errorf("expected failure %s for %v", errorCase.msg, errorCase.mode) </ins> } else if !strings.Contains(errs[0].Error(), errorCase.msg) { t.Errorf("unexpected error: %v, expected: %s", errs[0], errorCase.msg) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When running , and passing the flag which was introduced in , the following error occurs: What you expected to happen: The proxy should work as it used to, supporting this mode. How to reproduce it (as minimally and precisely as possible): ./kube- --proxy-mode=kernelspace Anything else we need to know?: This was introduced after the validation was refactored in which regressed validation for . IPVS validation (and thus support) was later , but we need another mode for Windows. Environment: Kubernetes version (use ): v1.10.0-alpha.0.684+ Cloud provider or hardware configuration: n/a OS (e.g. from ): Windows Server Core 1709 Kernel (e.g. ): Install tools: Others:
/sig windows /sig network
Tentatively adding milestone so we can track getting consensus on release-blocking status.
/priority critical-urgent
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: if len(name) == 0 { _, name, _ = scope.Namer.ObjectName(obj) } <del> admissionAttributes := admission.NewAttributesRecord(obj, nil, scope.Kind, namespace, name, scope.Resource, scope.Subresource, admission.Create, options, dryrun.IsDryRun(options.DryRun), userInfo) if mutatingAdmission, ok := admit.(admission.MutationInterface); ok && mutatingAdmission.Handles(admission.Create) { err = mutatingAdmission.Admit(ctx, admissionAttributes, scope) if err != nil { scope.err(err, w, req) return } } if scope.FieldManager != nil { liveObj, err := scope.Creater.New(scope.Kind) if err != nil { scope.err(fmt.Errorf("failed to create new object (Create for %v): %v", scope.Kind, err), w, req) return </del> <ins> trace.Step("About to store object in database") result, err := finishRequest(timeout, func() (runtime.Object, error) { if scope.FieldManager != nil { liveObj, err := scope.Creater.New(scope.Kind) if err != nil { return nil, fmt.Errorf("failed to create new object (Create for %v): %v", scope.Kind, err) } obj, err = scope.FieldManager.Update(liveObj, obj, managerOrUserAgent(options.FieldManager, req.UserAgent())) if err != nil { return nil, fmt.Errorf("failed to update object (Create for %v) managed fields: %v", scope.Kind, err) } </ins> } <del> obj, err = scope.FieldManager.Update(liveObj, obj, managerOrUserAgent(options.FieldManager, req.UserAgent())) if err != nil { scope.err(fmt.Errorf("failed to update object (Create for %v) managed fields: %v", scope.Kind, err), w, req) return </del> <ins> admissionAttributes := admission.NewAttributesRecord(obj, nil, scope.Kind, namespace, name, scope.Resource, scope.Subresource, admission.Create, options, dryrun.IsDryRun(options.DryRun), userInfo) if mutatingAdmission, ok := admit.(admission.MutationInterface); ok && mutatingAdmission.Handles(admission.Create) { if err := mutatingAdmission.Admit(ctx, admissionAttributes, scope); err != nil { return nil, err } </ins> } <del> } trace.Step("About to store object in database") result, err := finishRequest(timeout, func() (runtime.Object, error) { </del> return r.Create( ctx, name,

What happened: Currently, the mutating admission happens before updating managedFields in create requests, but in the update and patch handlers, mutating admission happens after updating managedFields. This causes the ownership of fields modified by mutating webhooks to be inconsistent based on the request verb. /sig api-machinery /cc /priority backlog
/wg apply
/assign

What happened: I noticed that in both the update and patch handler, the apiserver calls admission webhooks from within the asynchronous function, but in the create handler, we only call validating webhooks this way, and call the mutating admission chain from outside this asynchronous/timeout function () I think this could create inconsistent behavior in the case of a create request with a long running mutating admission step, since the timeout in finishRequest wouldn't include this part of the request /sig api-machinery /cc /priority backlog
/cc
I think the thinking is that Create doesn't need retries due to conflicts, so these hooks wouldn't need to be called multiple times. (but then why would the validating ones?) Honestly there is no good reason to have a finishRequest at all; it's a holdover from a very early long running operation concept that was removed.
until every single call in our request handling stack honors context deadline, I think we need it. we're getting closer, but I'm not positive we are quite there. at first glance, for this particular case, moving inside the function passed to finishRequest might be all that is needed


Question: start program = "/etc/init.d/docker start" stop program = "/etc/init.d/docker stop" if does not exist then restart <ins> if failed unixsocket /var/run/docker.sock protocol HTTP request "/version" then restart </ins>

During our Jenkins run, a cluster failed to initialize with a pattern like so: I logged into the second node to see what was going on, and found the kubelet logs seemed to be repeating themselves every couple of minutes: Checking monit, yup: Jenkins is currently disabled to allow live debugging of the issue. Raising to P0 until we get Jenkins back. cc
I can login to the node and working on it now.
Here is what happened: Docker daemon is hung during startup phase: But nono docker CLI calls return, including 'docker version'. Monit now does healthcheck on kubelet, which in turns to query docker version, which never response and timeout, and cause a kubelet restart. Looking into why docker daemon hang forever.
Ok, that docker daemon on a terrible state, and all docker API requests hang. This is known issue from docker since I personally reported twice. The current active one is docker/docker/issues/9605. 1) I tried to strace docker process for about 20 seconds, and nothing is interesting except one thread stucking at read syscall forever: 2) I sent SIGQUIT to docker daemon. Docker is shutdown, and restart by monit. From below log, one can notice that there is a huge gap between time="2015-03-11T17:58:25Z" and time="2015-03-11T21:06:06Z" 3) Monit now restart docker daemon: Assuming there are always docker API request hang issues, we should detect such issue, and restart docker like what I did for kubelet. Meanwhile, we shouldn't fail kubelet's healthcheck when docker API hang.
This is part of
Thanks for the quick fix!


Question: sudo docker ps ``` <del> And see the actual task.  (note that initial ```docker pull``` may take a few minutes, depending on network conditions. </del> <ins> And see the actual task.  (Note that initial ```docker pull``` may take a few minutes, depending on network conditions.) </ins> ### Step Two: Turn up the master service. A Kubernetes 'service' is named load balancer that proxies traffic to one or more containers.  The services in a Kubernetes cluster are discoverable inside other containers via environment variables.  Services find the containers to load balance based on task labels.  The task that you created in Step One has the label "name=redis-master", so the corresponding service is defined by that label.  Create a file named redis-master-service.json that contains:

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The image () used to run the per-pod infra/pause container logs to stdout about once per second. What you expected to happen: Pause container not logging anything. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Kubelet does not expose, nor manage these logs. Writing the logs wastes resources and takes up disk space, which may not be properly accounted by kubelet. The pause image used for the e2e tests seems much better: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign Will also rebuild with


Question: // storage medium is the default, then the volume is ready.  If the // medium is memory, and a mountpoint is present, then the volume is // ready. <del> if volumeutil.IsReady(ed.getMetaDir()) { </del> <ins> readyDir := ed.getMetaDir() if volumeutil.IsReady(readyDir) { </ins> if ed.medium == v1.StorageMediumMemory && !notMnt { return nil } else if ed.medium == v1.StorageMediumDefault { <del> return nil </del> <ins> // Further check dir exists if _, err := os.Stat(dir); err == nil { return nil } // This situation should not happen unless user manually delete volume dir. // In this case, delete ready file and print a warning for it. klog.Warningf("volume ready file dir %s exist, but volume dir %s does not. Remove ready dir", readyDir, dir) if err := os.RemoveAll(readyDir); err != nil && !os.IsNotExist(err) { klog.Warningf("failed to remove ready dir [%s]: %v", readyDir, err) } </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There is a node in my cluster whose disk is full on root partition, I cleared the disk and then rebooted the node to recover it. After the reboot, the kube-proxy on that node got stuck in "ContainerCreating" status and never changes, and all other pods can't run properly since the kubeproxy is not runing. What you expected to happen: Kubeproxy on that node can run normally after the disk clean and reboot How to reproduce it (as minimally and precisely as possible): the disk full on root partition in my env, leave roughly 100k space on root partition would make this issue more reproduciblethis issue is kind of tricky, to reproduce it, one need to populate the disk repeatedly since some disk space would be cleared because of pod evictionthe kube-proxy got deleted and created repeatedly because of pod eviction and daemon-set mechanism, it would stuck in ContainerCreating status if the disk space is not enough for creating volume contents of kube-proxy configMap volume, the first pod error event is like: MountVolume.Setup failed for volume "kube-proxy" space left on device After it got stuck, the following pod error event woud repeat over and over, kube-proxy is trying to mount configMap volume but always fail MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory the disk and reboot the reboot is neccesary to reproduce the issue, without the reboot the pod eviction would kick in to delete kube-proxy pod the kube-proxy woud stuck in ContainerCreating Status forever, restart kubelet has no effects on that, kube-proxy pod need to be mannully deleted to get recover kube-proxy pod can't mount configMap volume, the following error keeps repeating MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory Anything else we need to know?

: After digging into this issue for a quite long time, i think the reason for this issue is: The configMap volume uses emptyDir internally to create the volume directory, after the directory is created, configMap volume failed to populate it with configMap content because the disk is full, then configMap volume calls emptyDir.tearDown() to clear the temp directory. The problem is that emptyDir uses a "ready" file under /var/lib/kubelet/pods/{podId}/plugin/.. to determine whether the directory is created or not, if the ready file is present, then it wouldn't create the directory. The tearDown method only remove the directory, it doen't remove the ready file, so after the first out of disk error, the diretory creation always get skipped, hence no such file or directory error repeat over and over. Environment: Kubernetes version (use ): v.1.11.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): cent os 7 Kernel (e.g. ): Install tools: kubeadm Others:
/sig storage
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If it's any help, I just encountered this issue in kubelet v1.13.2 under Debian Stretch 9.5, after a restart. I was not out of disk space. Deleting the ready files from /var/lib/kubelet/pods/{podId}*/ helped.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen hit this just now with the same conditions as originally described, but without a reboot.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen </details
/reopen
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks ! this helped me get my cluster running after all pods were evicted and clearing space alone didn't fix the issue.


Question: } } <del> // This is the spec for the volume that this plugin wraps. var wrappedVolumeSpec = volume.Spec{ // This should be on a tmpfs instead of the local disk; the problem is // charging the memory for the tmpfs to the right cgroup.  We should make // this a tmpfs when we can do the accounting correctly. Volume: &api.Volume{VolumeSource: api.VolumeSource{EmptyDir: &api.EmptyDirVolumeSource{}}}, </del> <ins> func wrappedVolumeSpec() volume.Spec { // This is the spec for the volume that this plugin wraps. return volume.Spec{ // This should be on a tmpfs instead of the local disk; the problem is // charging the memory for the tmpfs to the right cgroup.  We should make // this a tmpfs when we can do the accounting correctly. Volume: &api.Volume{VolumeSource: api.VolumeSource{EmptyDir: &api.EmptyDirVolumeSource{}}}, } </ins> } func (b *configMapVolumeMounter) SetUp(fsGroup *int64) error {

Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Prow link to the 2 jobs: * Additional Context:
/sig storage cc
I'm working on , it's hard to get a job yaml right at the first time. I'll check iscsi. /assign
/triage accepted /kind failing-test
ci-kubernetes-e2e-storage-kind-disruptive fixed: has 1 green run pull-kubernetes-e2e-storage-kind-disruptive fixed: , it fails when it should fail, it's green otherwise, I'll continue with iscsi.
/reopen it's still red, but at least it installs now:
Reopened this issue. <detailsIn response to : /reopen it's still red, but at least it installs now: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
e2e-gce-iscsi-serial job passes now: e2e-gce-iscsi is getting greener: I'll let it run for few more days to see if today's failures are flakes. /reopen
Reopened this issue. <detailsIn response to : It's getting greener: /reopen </details
iscsi-serial is green:


Question: // PluginName is a string with the name of the plugin const PluginName = "ResourceQuota" <ins> var namespaceGVK = v1.SchemeGroupVersion.WithKind("Namespace").GroupKind() </ins> // Register registers a plugin func Register(plugins *admission.Plugins) { plugins.Register(PluginName,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It appears that validating admission requests for server-side apply are populating the for requests to create Namespace objects. Non server-side apply requests don't do this, and a missing namespace field is the only way to distinguish cluster-scoped resources from namespace-scoped resources. This appears to be a breaking behavioral change. See What you expected to happen: Requests to create namespaces would have a consistent signature between server-side apply and non-server-side apply contexts. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: kind OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig-api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/wg api-expression
/assign
/assign
Requests made to namespaced URLs (URLs which contain a segment) populate the attribute in AdmissionReview requests. The documentation of the AdmissionReview field is as follows: From Update/Patch/Delete requests to specific namespaces have consistently populate this attribute for modifications to Namespace objects. Server-side apply is a type of patch request, so even when it is creating an object, that attribute will be populated. The difference is caused by a difference in the URL used by POST and apply-based creates. Creation via POST requests are made against the endpoint, so no attribute was populated. Namespace computed from request path: Namespace admission attributes for POST-based creates: Creation via server-side apply is done via a PATCH request to , so the attribute is populated. Namespace computed from request path: Namespace admission for apply-based creates: This is currently working as intended. If we want to make the behavior consistent between the two, I'd consider populating the namespace attribute for POST-based creation of Namespace objects.

Thanks for the thorough analysis Jordan, the problem OP is trying to solve is that they want to know if the object is namespaced or not, and used the presence of the field to detect that. According to your analysis, that seems wrong. The typical way to know if an object is namespaced is to use the discovery information, would that be the right way to solve that?
They could use discovery, or can continue to use the presence of the attribute, special-casing requests (namespaces are the only cluster-scoped resource which can have the attribute populated in AdmissionReview requests).
the documentation should be clearer, and I'd consider updating the POST-based create behavior to include this information to be consistent with update/patch/delete requests
do you want to update the doc and/or the post-based create behavior?
Sure! I am happy to take care of both.
where do you think it makes sense to add the namespace to the post-based behavior? We could add it to , but that might interfere with authorization, right? Would it be better to add the namespace directly to the ?
The info from the request body is not available when parsing requestinfo, only data in the URL. That's sort of what I expected, around the point we do this to pull the name from the incoming object:
Makes sense, I will give it a go, thank you!
that will also require sweeping admission plugins that check the namespace attribute. examples:
I think that we don't need to change anymore now. Do you know if there are other places in the docs where I should check?
yeah, I think we're good now. thanks for helping identify and clean this up


Question: waitForDetachCallCount(t, 1 /* expectedDetachCallCount */, fakePlugin) } <ins> // Populates desiredStateOfWorld cache with one node/volume/pod tuple. // Has node update fail // Calls Run() // Verifies there is one attach call and no detach calls. // Marks the node/volume as unmounted. // Deletes the node/volume/pod tuple from desiredStateOfWorld cache. // Verifies there are NO detach call and no (new) attach calls. func Test_Run_Negative_OneDesiredVolumeAttachThenDetachWithUnmountedVolumeUpdateStatusFail(t *testing.T) { // Arrange volumePluginMgr, fakePlugin := volumetesting.GetTestVolumePluginMgr(t) dsw := cache.NewDesiredStateOfWorld(volumePluginMgr) asw := cache.NewActualStateOfWorld(volumePluginMgr) fakeKubeClient := controllervolumetesting.CreateTestClient() ad := operationexecutor.NewOperationExecutor(fakeKubeClient, volumePluginMgr) nsu := statusupdater.NewFakeNodeStatusUpdater(true /* returnError */) reconciler := NewReconciler( reconcilerLoopPeriod, maxWaitForUnmountDuration, dsw, asw, ad, nsu) podName := "pod-uid" volumeName := api.UniqueVolumeName("volume-name") volumeSpec := controllervolumetesting.GetTestVolumeSpec(string(volumeName), volumeName) nodeName := "node-name" dsw.AddNode(nodeName) volumeExists := dsw.VolumeExists(volumeName, nodeName) if volumeExists { t.Fatalf( "Volume %q/node %q should not exist, but it does.", volumeName, nodeName) } generatedVolumeName, podAddErr := dsw.AddPod(types.UniquePodName(podName), controllervolumetesting.NewPod(podName, podName), volumeSpec, nodeName) if podAddErr != nil { t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podAddErr) } // Act go reconciler.Run(wait.NeverStop) // Assert waitForNewAttacherCallCount(t, 1 /* expectedCallCount */, fakePlugin) verifyNewAttacherCallCount(t, false /* expectZeroNewAttacherCallCount */, fakePlugin) waitForAttachCallCount(t, 1 /* expectedAttachCallCount */, fakePlugin) verifyNewDetacherCallCount(t, true /* expectZeroNewDetacherCallCount */, fakePlugin) waitForDetachCallCount(t, 0 /* expectedDetachCallCount */, fakePlugin) // Act dsw.DeletePod(types.UniquePodName(podName), generatedVolumeName, nodeName) volumeExists = dsw.VolumeExists(generatedVolumeName, nodeName) if volumeExists { t.Fatalf( "Deleted pod %q from volume %q/node %q. Volume should also be deleted but it still exists.", podName, generatedVolumeName, nodeName) } asw.SetVolumeMountedByNode(generatedVolumeName, nodeName, true /* mounted */) asw.SetVolumeMountedByNode(generatedVolumeName, nodeName, false /* mounted */) // Assert verifyNewDetacherCallCount(t, true /* expectZeroNewDetacherCallCount */, fakePlugin) verifyNewAttacherCallCount(t, false /* expectZeroNewAttacherCallCount */, fakePlugin) waitForAttachCallCount(t, 1 /* expectedAttachCallCount */, fakePlugin) verifyNewDetacherCallCount(t, false /* expectZeroNewDetacherCallCount */, fakePlugin) waitForDetachCallCount(t, 0 /* expectedDetachCallCount */, fakePlugin) } </ins> func waitForNewAttacherCallCount( t *testing.T, expectedCallCount int,

introduced logic to make kubelet volume manager wait for attach detach controller to finish attach before it starts mount. The controller should also not start a detach operation until the node status has been updated removing the volume from the list of attached volumes. Otherwise if a pod is rapidly created and deleted on the same node, it could result in a mount starting immediately before a detach starts. Props to and his pet set tests for helping catch this.
Hmm, this just happened , just a lot less frequently. Haven't debugged, but it looks like the failed pod did reland on the same node: pet-0, jenkins-e2e-minion-group-4mn2 (). Reopening, will try to confirm when i have time, and close if it's not the same thing.
Just following up with an update, Saad and i are tag teaming this. He's uploading a pr that I'm going to reivew. We're tracking a couple of issues: need to update the "volumes in use" list before polling on the "volumes attached" list. Best option is to write a operation executor, so we don't hammer the apiserver. needs to get up to date node object from apiserver to determine volumes in use. The update that removes volumes from the "attached list" shoud return a fresh node object, we can just iterate the volumes its using (this must happen after popping from attached, though). leaky volumes in the node object (this is not about the physical volumes but about the size of the json blob that's the node object growing). The proposed fix is to use the desired state of the world to update the list of volumes in use. There's a last point I need to follow up on. We might leave the node object in an inconsistent state with the proposed fix, but it should be ok since nothing is checking the inconsistent fields. This will happen if we pop a volume from the attached list, but then find its in use in the updated node object. ping me when you're ready to go, of if you need help


Question: set -o errexit set -o nounset <del> set -o pipefailscript_dir=$(cd $(dirname "${BASH_SOURCE}") && pwd -P) </del> <ins> set -o pipefail script_dir=$(cd $(dirname "${BASH_SOURCE}") && pwd -P) </ins> cd "${script_dir}"

after was merged It seems to be related to the machine type. /cc /sig scalability k8s-infra
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
I'll fix this now


Question: healthz.InstallHandler(s.restfulCont, healthz.PingHealthz, healthz.NamedCheck("docker", s.dockerHealthCheck), <del> healthz.NamedCheck("hostname", s.hostnameHealthCheck), </del> healthz.NamedCheck("syncloop", s.syncLoopHealthCheck), ) var ws *restful.WebService

When looking at Kubelet healthz through proxy on master I'm getting: cc
cc
kubelet expects to be the hostname, while master uses IP in the request instead. /cc people who knows more about proxy:
Master's expectation looks wrong.
The idea of that health check is to ensure that the master schedules work to the name expected by the kubelet. Otherwise sometimes you end up in a situation where the master is scheduling work for IP and the kubelet is looking for work for 'hostname -f'; the node status says 'Ready' but no pods get started on that node. User's must use --hostname_override to correct the situation. It has been a while since I have looked at that code so things may have changed, but in this case is the master scheduling using the hostname "10.240.170.229" ? If so then pods will not start on that node. If not then the health check needs a better way of getting the hostname as the master knows it.
I don't think this is related to the proxy, I think your master has an incorrect node configuration, as says.
- I checked this on and clusters as well, with the same result. All of us were running default GCE configuration.
That's very weird, I reproduce, too. But it has nothing to do with the proxy, rather it's the requested host (????!?): So kubelet gives different answers depending on whether you refer to it as "localhost" or by its IP address.
If kubelet is doing the healthz check about the host in the HTTP request, then that's wrong. To be useful, it should be making an assertion about the Node object that represents it in the system.
To give more context, the change was introduced in I think was describing a situation where kubelet and master have discrepancy on what the node is. E.g., kubelet thinks it's node HOSTNAME, while master thinks the node is IP. In this case, kubelet wouldn't know which Node object to check. if I understand correctly, this should not happen now because kubelet would not update the node status of the incorrect node, so the node would eventually become unavailable kubelet can register itself now
Yeah the original patch in sent the host name presumed by master along with the request but we though the hostname from the header sufficed as we could not come up with a scenario where the two differ.

That can be changed back if needed. It looks like the localhost bypass was later in in your cluster if you disable the health check do pods actually start on the node ?
hmm... do you mean the master would not be able to reach the node if it has the incorrect name ? The hostname is correct it is just different from what the node uses. Say for example the host uses "" and the node node knows itself as "93.184.216.34". The health check will pass because the node is reachable, but pods don't start because work no work is scheduled for "93.184.216.34" Yeah I think that is a safe path.
kubelet updates node status now, and if the node status isn't updated for sometime, node controller would deem the node unhealthy. E.g., if the node knows itself as "93.184.216.34", it'd try to update the status of node "93.184.216.34". The status of node "" would not be updated by anyone, and would eventually become unhealthy.
It fails through the proxy because master's proxy code uses IP address (not hostname) to reach the node. Master is actually doing the right thing there. Master doesn't directly health check kubelets any more (monit does) so that's why this isn't currently breaking anything. I'm guessing the localhost exception was because that's how monit health checks; but that's evidence that the original assumption that one could use the HTTP request's host for this was faulty, and the check should be completely removed or done some other way.
got it! I see. The check is no longer correct then and given what explained the check is unnecessary :) I'll open a PR to remove it.
thanks!


Question: } } <ins> func TestWatchCacheBypass(t *testing.T) { backingStorage := &dummyStorage{} cacher, _, err := newTestCacher(backingStorage) if err != nil { t.Fatalf("Couldn't create cacher: %v", err) } defer cacher.Stop() // Wait until cacher is initialized. if err := cacher.ready.wait(); err != nil { t.Fatalf("unexpected error waiting for the cache to be ready") } // Inject error to underlying layer and check if cacher is not bypassed. backingStorage.injectError(errDummy) _, err = cacher.Watch(context.TODO(), "pod/ns", storage.ListOptions{ ResourceVersion: "0", }) if err != nil { t.Errorf("Watch with RV=0 should be served from cache: %v", err) } // With unset RV, check if cacher is bypassed. _, err = cacher.Watch(context.TODO(), "pod/ns", storage.ListOptions{ ResourceVersion: "", }) if err != errDummy { t.Errorf("Watch with unset RV should bypass cacher: %v", err) } } </ins> func TestWatcherNotGoingBackInTime(t *testing.T) { backingStorage := &dummyStorage{} cacher, _, err := newTestCacher(backingStorage)

As Jordan pointed out in: watchcache treats RV=0 and RV="" the same way, even though it's not what we document: This should be fixed. /kind bug /sig api-machinery /priority important-soon
/assign
/triage accepted


Question: podStatus.Host = kl.GetHostname() hostIP, err := kl.GetHostIP() if err != nil { <del> return api.PodStatus{}, fmt.Errorf("Cannot get host IP: %v", err) </del> <ins> glog.Errorf("Cannot get host IP: %v", err) } else { podStatus.HostIP = hostIP.String() </ins> } <del> podStatus.HostIP = hostIP.String() </del> return *podStatus, nil }

I suspected this is a regression introduced by . I observed a pod was restarted and stuck at PENDING phase for a long time (Don't know the reason why it is restarted at the first place, but that is a separate issue) ssh to the machine, I found both containers are running happily on the machine: Checked kubelet's log, I found kubelet bailed out on generating PodStatus because the failure of retrieving HostIP: I didn't look into with more detail yet. If no one pick up this tonight, I will take a look tomorrow morning.
cc
That' interesting. kubelet bug comes from GetHostIP() function. It reads cached Node and parse it's addresses. Apparently Nodes does not have any addresses reported in status field. Is that expected? Pod_cache just ignored missing HostIP, so I'll send a PR to have a similar logic in kubelet.
I sync-ed, but still observed the same issue: Here is the error from
The issue here is that NodeStatus.Address is empty. I am closing this one, and filing a separate one for cloud provider.
I'm seeing this problem with v0.15.0. What is the new tracking issue?
at the end I didn't file a separate issue because I couldn't reproduce the issue later. Feel free to file a new issue with detail please.
I'm also seeing this, but don't have a repro case yet unfortunately. what did you mean when you said you sync-ed?
- where are you running your cluster, and how did you create it?
I’m running the Master related process in one Docker container and the Node related process in another Docker container. All the Nodes are connected using OpenVSwitch GRE tunnels. The Nodes register with the Master using this script
I am receiving this same error running master and node on separate coreos images with vagrant on parallels. I am able to register the node, but then health check fails and results in this error.