Question: // TODO(dchen1107): Change Info to list from map podStatus.Info = info <del> return podStatus, err </del> <ins> return podStatus, nil </ins> } // Returns logs of current machine.

I have a cluster with ~130 pods. Running back-to-back 'get pods' shows me wildly different understandings of their state, even though I can tell from docker that they have not crashed or anything. thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 127 Running 1 STATUS 6 Unknown thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 82 Running 1 STATUS 51 Unknown Nothing changed in between these calls.
cc I blame the pod cache.
has recently changed pod cache? FWIW, I also observed this behavior: 19 pods Running in one call and all Unknown in the next. With 0.10.0.
Yes, I think PR on pod cache should fix the problem reported here. Also it should fix the issue reported by about adds-on services. I have a pending PR to have kubelet deciding on PodStatus too.
Could this have caused ?
I don't see how. I'll comment on that issue.
I don't think this could be the cause of that one.
I don't think PR on pod cache will fix this. That one clears the node status. This issue is with pod going from Running to Unknown even when nodes keep running just fine.
Totally consistent observation based on my personal experience. I just assumed we were shooting for some kind of Hesinberg Uncertainty Principle. S , Tim Hockin wrote:
Agree with clears node status only. One possibility is that we hit transient network problem in between the calls that fails health check, so controller set node condition to not reachable, hence unknown pods. I'll make this more robust anyway. Other than that, it's hard to think of a reason that node status results in the flakiness.
In the absence of node information, the pod status defaults to Unknown. Maybe there's another source of this specific problem, but the pod cache is a source of continuous problems, partly because it's not just a cache, but is implementing or duplicating a significant chunk of system logic in a half-baked manner. My goal is to eliminate it -- see .

Short of that, we need to at least implement , eliminate nodes from the pod cache, and implement a real node cache.
I can look into this. Repro info from Tim, who said he was running on a one-node cluster but has seen the problem with multi-node clusters also: run-container hostnames --image=kubernetes/servehostname --replicas=1 --port=9376 --labels=app=hostnames expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" resize --replicas=150 rc hostnames but I had to hack the node a bit to make it get past 100 on the node: sysctl -w fs.inotify.maxuserinstances=8192 echo "DOCKERNOFILE=" /etc/default/docker; /etc/init.d/docker restart (fails sometimes, dunno why)
I can verify that the situation is rather bad. Lots of fluctuation in the numbers of Pending, Running, and Unknown. serve_hostname containers on the node are exiting; I'm not sure why, or if the pod itself is terminating or just the container. Lots of the following in the log (not sure what this means yet): Sat, 07 Feb 2015 00:06:22 +0000 hostnames-ecd2r Pod failedScheduling {scheduler } Binding rejected: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
The pause container is also exiting, so I'm guessing that means the whole pod is exiting, not just serve_hostname...
Ugh, there is an etcd size limit you might be hitting too, I forgot. If you are seeing the fluctuations, maybe you got far enough, though. On Feb 6, 2015 5:13 PM, "David Oppenheimer" wrote:
Just a few more observations: This is reproducible (and a little less chaotic) with one node and 100 pods (but not with 50 pods, so it's definitely load-dependent). You can get to 100 Running pods (0 Pending or Unknown), and then a minute later a dozen or so will become Unknown, and then later go back to Running, etc. As mentioned earlier, the containers continue running on the node all this time (sudo docker ps wc -l), so it is definitely some kind of reporting problem.

As an aside, it looks like there is some kind of race condition with IP address allocation, as during startup (but not at steady state) a number of pods exit with the following in their log (sudo docker logs xxx): listen tcp :9376: bind: address already in use They do restart OK (I'm not sure if this is being restarted locally on the node or making a round-trip back to the scheduler).
Almost certainly the restart is just kubelet retrying after some time. On Feb 8, 2015 2:57 PM, "David Oppenheimer" wrote:
I found that if I comment out the line with the arrow next to it (see below) in , the Pods stop oscillating between Running and Unknown. Commenting out this line changes the logic from "if get an error from Kubelet, change pod status to Unknown" to "if get an error from Kubelet, keep pod status unchanged." I saw in the master logs intermittent errors of this form, which was causing the pods to flip back to Unknown: E0209 04:19:17. 4996 ] Error getting info for pod default/hostnames-v5drx: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-v5drx This seems to correspond to lines like this in the kubelet logs: I0209 06:38:11. 4704 ] GET /stats/default/hostnames-gfw5j/-b026-11e4-a070-/hostnames?num_stats=33: (218.857218ms) 500 goroutine 730 [running]: [ ... stack trace, but they're not always the same; I'll dig into this some more, the default stack trace depth wasn't going back past the various functions that just serve up the error ... ] However, pods still flip between Running and Pending, even though no containers are exiting on the minion, so there's another problem here too.
Thanks for investigating this! Pods really shouldn't flip between Running and Pending, as I believe that the only way that they go Pending is if Pod.Status.Host is empty, and this really shouldn't happen post scheduling, unless we are somehow dropping information. Maybe we need to make sure that we don't wipe out status on the error case?

--brendan , David Oppenheimer
I also investigated the same issue last Friday. I observed exact the same as what described above, except the related kubelet log should be: Meanwhile, I observed master is trying to query pod / and failed all the time: Talked to offline, he is going to look into why running pods go back to Pending; meanwhile, I am going to figure out Unknown cases here.
That would be great. It is very hard for me to make sense of any of the scalability experiments that I am doing while the pod status is in hysteresis.
Just to summarize: (1) the Running<-Unknown flapping is caused by the line of code I mentioned earlier; (2) the reason that line of code is being hit on the master is due to two errors on the kubelet (the one I identified earlier was wrong; Dawn found these): I0209 06:42:09. 4704 ] GET /api/v1beta1/podInfo?podID=hostnames-v5drx&podNamespace=default: (8.82543048s) 500 E0209 04:34:13. 4996 ] Error getting info for pod default/hostnames-2y8ny: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-2y8ny (3) we don't yet know the cause of the Running<-Pending flapping; I'm going to look into that now
(and Dawn is looking into why we're hitting those two errors on Kubelet)
Are we serving requests concurrently in Kubelet? (Is it possible not to when using go's servemux?) I don't see any locking around the pod map accesses.
This was marked for the fixit but Dawn and I are working on it. Assigned to myself.
I am also investing time writing code to try and report what is going on with unstable cluster status. It would be good to know what you and Dawn are doing in case I am wasting my time. Thanks. , David Oppenheimer
This particular problem is easily reproduced manually using the instructions Tim listed, so I'm not sure it's necessary to write anything sophisticated to drive the cluster into this state.

Anyway, maybe give me and Dawn a couple of days and if we can't figure this out by then we can call in the cavalry.
Am I wrong, or are these instructions also within bounds of an e2e test? Just because our existing e2es only do one or two things doesn't mean we can't do more within O(minutes), assuming you're dealing with the same image.
The "flips back to pending" problem appears to be because updatePodStatus() writes to the cache but not back to the Pod's status. computePodStatus() initializes newStatus based on pod.Status, not based on the cached value of status. This second behavior seems wrong in light of the first behavior. However, I believe this only causes a problem in combination with my fix described earlier, which as a result doesn't actually do what I described (it doesn't "keep the status as-is" but rather copies it out of the pod's status; to "keep the status as-is" it should be copying it out of the cache). I can see if this fixes things. (We also should fix the root cause stuff on the node side, which Dawn is working on.)
Assign the issue to me ... I didn't see the flips between Pending and Running anymore. Don't know why. I consistently see flips between Unknown and Running on those heavily loaded nodes. The root issue is once a while docker inspect API call returns us EAGAIN. Please note that on my test nodes, I already increase NOFILE limit for docker daemon process. The proposed solution is that kubelet deals with all errors internally except EAGAIN one. In this case, computePodStatus logic in pod_cache does nothing when p.containerInfo.GetPodStatus returns an error. Working on a PR to fix this.
Dawn, my suggestion is that computePodStatus() should return the current cached value whenever it gets an error from Kubelet. I can implement this if you want.
, why shouldn't updatePodStatus simply write through to the pod's status? I'd think either we always read the cached value first, or we always write through to the pod and read from there. Only reading the cached value upon an error seems counter-intuitive to me...
why do you computePodStatus returns a cached value here?

The only place calls computePodStatus is updatePodStatus, which triggered by two operations: 1) periodically update all containers 2) the client asks for PodStatus, and there is no cached one, instead calling computePodStatus to retrieve one. I think a cleaner way is 1) Kubelet handles all the errors except this transient one. Of course, we need to make sure docker is auto-restart and there is no too many open files error on node side too. But they are separate issues and should be handled by node side through script or configure etc. 2) Now back to master side, if node status is ok, kubelet should return something; if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round. I have one working PR to avoid the flips between Running and Unknown, but with increasing number of Pending from node side caused by docker failure. Still investigating it...
I didn't understand "if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round." If you're talking about the caching in the podCache on the master, then what you wrote is exactly what I was suggesting. (If you're talking about some kind of caching on the Kubelet, then that's different, because I am not aware of cache for status on the Kubelet.) I think always writing back to the pod's status might be fine but I don't know why the podCache was designed the way it was, so I didn't want to change that without asking Daniel first (I think he's the one who created it). I wasn't suggesting to only read the cached value upon an error; I meant always read the cached value under all circumstances.
and here is my understanding. Correct me if I am wrong. UpdateAllContainers periodically updates all containers, but it gets the initial pods from etcd through this line . each from , UpdateAllContainers calls passes to . computes and returns a new status. Note that the status is computed based on , i.e., the information in etcd. updates the pod cache with the new status.

Note that the new status is not written to etcd. If you repeat the same steps again, you will start with the stale pod status in the etcd because does not check the cached value.
I have tested since last night. The flaps between Running, Pending and Unknown are gone for about 10 hour running 300 pods on 4 nodes except two scenario: 1) etcd issue: cat kube- | grep "status to unknown" I0212 20:18:30. ] node doesn't exist: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] <nil, setting pod hostnames-yuy8d status to unknown ... This happens a couple of times last 10 hours. We need to figure out why. 2) NodeStatus becomes UNREADY. It happens once last 10 hours, and pods status on that node become Unknown which is expected. I didn't figure out why NodeStatus was UNREADY then, but I heard there is a network earlier this morning.
up to you whether to reopen this issue or file new ones for new issues.
I've also seen the "All the given peers are not reachable" message (see my second entry in this issue). I think that was what was referring to when he said "there is an etcd size limit you might be hitting too" but I may be mis-interpreting.
That error is the etcd client's catch-all for "something went wrong" - it's actually really bad that it obscures errors. , David Oppenheimer
, Tim Hockin wrote:
and Yes, that error message is cache-all error message, which need to investigation. In my particular case, it shouldn't be etcd size limits at all since we have that cluster built and upgraded more than 10 hours before, and all 300 pods were scheduled back then. There is no new events generated either from kubelet or other components since all bound pods are indeed happily running on the nodes. No new writes to etcd except NodeStatus, which is very less frequent and not much data. The problem of etcd here occurs when master tries to read from etcd with an error.


Question: } func (ed *emptyDir) teardownDefault(dir string) error { <del> tmpDir, err := volume.RenameDirectory(dir, ed.volName+".deleting~") if err != nil { return err } err = os.RemoveAll(tmpDir) </del> <ins> // Renaming the directory is not required anymore because the operation executor // now handles duplicate operations on the same volume err := os.RemoveAll(dir) </ins> if err != nil { return err }

While debugging a node-e2e failure on Ubuntu image, I noticed that the kublet log is full of the following error messages from: The above failure results in tons of EmptyDir volumes (including both secret and confgmap) are not properly deleted, thus test failures. cc/
CC
could you share the complete kubelet logs please
you can access the kubelet log from previous run at: They have the same issue.
for the most recent test run see also
Is there a link not behind a login?
Here is one of the errors: Saad and I debugged. Our hypothesis is that: The Rename() method for EmptyDir is first creating a directory with a new name and then tries to rename the old directory to the new name. That os.Rename() is failing on Ubuntu because the directory already exists, which succeeds on other images. To verify the Ubuntu issue, we can write a small test program using os.Rename() directly on the Ubuntu image. Then the volume reconstruction logic is picking up the new directory as a new emptydir volume for the pod, which triggers another deletion round.
Some things we should improve regardless of the root cause of this: volume reconstruction logic should ignore any folders (to break the explosion of directories/volumes in cases like this). the the default method for : Understand why it does a create directory before rename. Eliminate the create directory if possible. Otherwise make sure that the created directory is deleted if subsequent rename fails. Understand why a rename is required at all before delete. Eliminate it if possible. Otherwise make sure that if delete fails, that the rename is undone.
BTW, in Go 1.8, it looks like they've changed the behavior of os.Rename: So I think we should move towards eliminating the create directory.
Does this mean the problem could be caused by a difference in the use of golang 1.7 vs. golang 1.8 for k8s binary builds?
Yes that's possible. From the development guide, go1.8 has not been verified for building k8s.
related
Is the test binary which generates the failures reported here is built with golang 1.8? I couldn't find those information through the existing logs.
From what I understand, the purpose of the rename is so that the system knows quickly that the volume has been deleted.

This is important for volumes with large content that can take a long time to delete. So I think the right set of changes that are needed are: The volume reconstruction logic should ignore directories with ".deleting~". I can make this change. The rename needs to be changed to not create a directory first. It looks like is handling this.
On Thu, Mar 23, 2017 at 01:27:30PM -0700, Dawn Chen wrote: Yes, it is. Should we build with golang-1.7 instead?
Confirmed that when building with go1.7 instead of go1.8, the node tests succeed.
Great! Thanks for the updates. We can continue carrying on our integration work. And the issue has nothing to do with Ubuntu image itself. I am leaving this open to fix the error-prone logic in EmptyDir's teardown logic, and resolving the golang 1.8 compatibility issue.
looks like a possible dup of
I've encountered this issue about a month ago and now I can confirm that this issue is caused by compiling with Go 1.8, which forbids renaming a directory into an existing directory even if the destinition is empty. So the current logic that uses call does not work properly under Go 1.8. The correct way to solve it is to change the behavior in so that it does not rely on the call to which creates a new empty directory after a successful call.
I discussed with Saad. We think that after the volume manager redesign, we don't need to do a rename at all. The volume operation executor can handle duplicate requests for volumes with the same name. I will work on testing a fix for this.


Question: <del> // +build linux </del> /* Copyright 2017 The Kubernetes Authors.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Currently, IPVS proxier only runs in linux platfrom. What you expected to happen: Run IPVS proxier UTs everywhere, including !linux platfrom, which will help a lot when developing in windows platfrom. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig network
/area kube-proxy


Question: "strings" "testing" <del> "k8s.io/component-base/metrics/legacyregistry" </del> <ins> "k8s.io/component-base/metrics" </ins> "k8s.io/component-base/metrics/testutil" )

Change encryption config controller reload metrics to be consistent with other places (Authn, Authz) xref: Combine and into a single metric with with an enum value of and . /sig auth /triage accepted
cc
We probably should keep the existing metrics for some time to allow folks to migrate to the new one.
Ah I was thinking about this as well! Looking at "may be deleted at any time", I was leaning towards just removing it. If the common approach is to keep it for a release before removing, we could do that. I believe we can set a for the 2 metrics we want to remove.
I don't have a good sense of metrics stability because they tend to stay alpha forever and folks don't go back to update them. Since the associated feature set is GA, I think we should be conservative with the changes we make. Perhaps SIG instrumentation folks can provide some guidance here?
As part of the PR, I was going to ask when is the right time to graduate these metrics do you have thoughts on to take while deleting an alpha metric metrics should be graduated from alpha to beta and stable?


Question: return &Client{client}, nil } <ins> // MatchesServerVersion queries the server to compares the build version // (git hash) of the client with the server's build version. It returns an error // if it failed to contact the server or if the versions are not an exact match. </ins> func MatchesServerVersion(c *Config) error { client, err := New(c) if err != nil {

This seems like a bug to me No output, and command exits success... wat. Have to explicitly pass to see expected output. If the client version is greater the server version, it should be making requests based on the server's newest accepted apiversion.
cc


Question: package main import ( <ins> "errors" </ins> goflag "flag" "fmt" "math/rand"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: when i run hollow-node deploymenthollow-proxy will cause panic. hollow-node deployment is as follows: hollow-proxy container logs: I1012 12:37:13. 7 ] syncProxyRules took 622.278µs I1012 12:37:13. 7 asmamd64.s:573] Shutting down service config controller E1012 12:37:13.

7 ] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) /usr/local/go/src/runtime/asmamd64.s:573 /usr/local/go/src/runtime/asmamd64.s:2361 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2c68ad9] goroutine 74 [running]: (0x0, 0x0, 0x0) +0x107 panic(0x32dc9e0, 0x5c20d20) +0x229 (FakeExec).LookPath(0xc420ad4450, 0x3ac6dba, 0x9, 0xc4208c51d0, 0x0, 0xc4200a4800, 0x4e7e83) +0x39 (0x3e3c000, 0xc420ad4450, 0xc4208c51d0, 0x5, 0x5, 0x5, 0x5) +0x5e (0x3e3c000, 0xc420ad4450, 0xc420768c70, 0xc, 0x3abdd6d, 0x3, 0x0, 0x0) +0x17d (Proxier).syncProxyRules(0xc420d8fe40) +0xb643 (Proxier).OnServiceSynced(0xc420d8fe40) +0x62 (ServiceConfig).Run(0xc420ad5080, 0xc4200b62a0) +0x183 created by (*ProxyServer).Run +0x283 What you expected to happen: How to reproduce it (as minimally and precisely as possible): boot up hollow-proxy container with "--use-real-proxier=true" Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g.

): Install tools: Others: I found that in "kubemark/hollow-" we init a 'FakeExec' struct, but we don't init LookPathFunc in 'FakeExec' struct. So, execer.LookPathFunc is a vild pointer, so it cause panic.
/sig Testing
maybe we need init:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale I met this issue as well.
/sig scalability


Question: <ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package apiserver import ( "encoding/json" "fmt" "io/ioutil" "log" "net" "net/http" "strconv" "github.com/GoogleCloudPlatform/kubernetes/pkg/health" ) // TODO: this basic interface is duplicated in N places.  consolidate? type httpGet interface { Get(url string) (*http.Response, error) } type server struct { addr string port int } // validator is responsible for validating the cluster and serving type validator struct { // a list of servers to health check servers map[string]server client  httpGet } func (s *server) check(client httpGet) (health.Status, string, error) { resp, err := client.Get("http://" + net.JoinHostPort(s.addr, strconv.Itoa(s.port)) + "/healthz") if err != nil { return health.Unknown, "", err } defer resp.Body.Close() data, err := ioutil.ReadAll(resp.Body) if err != nil { return health.Unknown, string(data), err } if resp.StatusCode != http.StatusOK { return health.Unhealthy, string(data), fmt.Errorf("unhealthy http status code: %d (%s)", resp.StatusCode, resp.Status) } return health.Healthy, string(data), nil } type ServerStatus struct { Component  string        `json:"component,omitempty"` Health     string        `json:"health,omitempty"` HealthCode health.Status `json:"healthCode,omitempty"` Msg        string        `json:"msg,omitempty"` Err        string        `json:"err,omitempty"` } func (v *validator) ServeHTTP(w http.ResponseWriter, r *http.Request) { reply := []ServerStatus{} for name, server := range v.servers { status, msg, err := server.check(v.client) var errorMsg string if err != nil { errorMsg = err.Error() } else { errorMsg = "nil" } reply = append(reply, ServerStatus{name, status.String(), status, msg, errorMsg}) } data, err := json.Marshal(reply) log.Printf("FOO: %s", string(data)) if err != nil { w.WriteHeader(http.StatusInternalServerError) w.Write([]byte(err.Error())) return } w.WriteHeader(http.StatusOK) w.Write(data) } // NewValidator creates a validator for a set of servers. func NewValidator(servers map[string]string) (http.Handler, error) { result := map[string]server{} for name, value := range servers { host, port, err := net.SplitHostPort(value) if err != nil { return nil, fmt.Errorf("invalid server spec: %s (%v)", value, err) } val, err := strconv.Atoi(port) if err != nil { return nil, fmt.Errorf("invalid server spec: %s (%v)", port, err) } result[name] = server{host, val} } return &validator{ servers: result, client:  &http.Client{}, }, nil } func makeTestValidator(servers map[string]string, get httpGet) (http.Handler, error) { v, e := NewValidator(servers) if e == nil { v.(*validator).client = get } return v, e } </ins>

If the scheduler is not healthy, pods won't schedule and things will get stuck in pending. Right now, there is no user visible feedback (in logs or events or pod status) that indicates that this is true. We should really have at least some logging in the apiserver, and possibly more in-depth information. I'm thinking that we should add a /validatez handler to the apiserver that does a quick sanity check. Added bonus: we could rip out all of the scripted health checking in the startup scripts.
A simple status HTTP page would be useful, showing the health status of each system component. We should seriously think about self-hosting, too. It doesn't seem like it would be hard to run a couple scheduler replicas using master election, managed by a replication controller.
Yeah, I think system components should register themselves with apiserver.


Question: } func getTestPod(critical bool, name string, resources v1.ResourceRequirements) *v1.Pod { <ins> value := scheduling.SystemCriticalPriority </ins> pod := &v1.Pod{ TypeMeta: metav1.TypeMeta{ Kind:       "Pod",

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Setting PodPriority directly when adminssion Priority enable,the Pod is rejected. To fix this, we need to create new priority classes for these tests. Which jobs are failing: Which test(s) are failing: I0822 19:01:41.165] should be able to create and delete a critical pod [It] Since when has it been failing: Testgrid link: Reason for failure: because adminssion Priority enableIf the priority class is not found, the Pod is rejected. Anything else we need to know:
/area e2e-test-framework /sig testing /kind bug /priority important-soon
This has now changed to this error: /reopen
Reopened this issue. <detailsIn response to : This has now changed to this error: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For reference: testgrid link /milestone v1.17
is this still in progress?
pr has updated, PTAL.
looks like has to merge for this to close
/milestone v1.18


Question: } func (g *glusterFSDriver) CreateDriver() { <del> // Reset config. It might have been modified by a previous CreateVolume call. g.driverInfo.Config.ServerConfig = nil </del> } func (g *glusterFSDriver) CleanupDriver() {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce/gke multizone/regional jobs Which test(s) are failing: Only PD tests Since when has it been failing: Around 12/21 with the 16:49 run Testgrid link: https://k8s- Reason for failure: TBD Anything else we need to know: <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
The runs started flaking after merged. cc Here's one failing run: Pod pod-subpath-test-gcepd-dynamicpv-pfsm failed to schedule because of conflicting node selectors: We shouldn't need to set the node selector for PD tests using PVs.
The PR broke the subpath test because the modification to the config struct embedded in the driver info now persists across tests: A quick fix would be to reset that field in CreateDriver. The long-term fix is the change discussed in My preference is to do the quick fix now, then either merge or close PRs in this order: (needs a decision about logging of tests that aren't getting defined, see ) a future PR for What I'd like to avoid is having to rebase PR on top of the long-term solution for issue - that'll be lots of code conflicts.
/assign PR is here: PR


Question: } loadBalancerIP := apiService.Spec.LoadBalancerIP <del> if loadBalancerIP != "" { </del> <ins> if loadBalancerIP != "" && internalAnnotation { </ins> createOpts.Address = loadBalancerIP }

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently openstack cloud provider just support for specifying internal LoadBalancerIP. I think that is unreasonable, actually user just use a floating ip to access a LoadBalancer service. So it is necessary to spport for specifying external LoadBalancerIP. What you expected to happen: Support for specifying external LoadBalancerIP on openstack Anything else we need to know?: Currently LoadBalancerIP can be specified in service.Spec.LoadBalancerIP, we can add annotation to distinguish between internal and external.
/sig openstack /assign I'd like to work for it.


Question: break } } <del> dropDisabledFields(crd, nil) </del> } // PrepareForUpdate clears fields that are not allowed to be set by end users on update.

This is the tracking issue of promoting CRD validation rules to GA: Listed the blocking issues/features below: [x] [x] [x] [x] Issues be aware of but should not be a GA blocker [ ] [ ] [ ] [ ] [ ] /sig api-machinery /triage accepted
/assign
/cc


Question: MarkVolumeAsUnmounted(podName volumetypes.UniquePodName, volumeName v1.UniqueVolumeName) error // Marks the specified volume as having been globally mounted. <del> MarkDeviceAsMounted(volumeName v1.UniqueVolumeName) error </del> <ins> MarkDeviceAsMounted(volumeName v1.UniqueVolumeName, devicePath string) error </ins> // Marks the specified volume as having its global mount unmounted. MarkDeviceAsUnmounted(volumeName v1.UniqueVolumeName) error

We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package apimachinery import ( "fmt" apiextensionsv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1" "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset" "k8s.io/apiextensions-apiserver/test/integration/testserver" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/watch" "k8s.io/client-go/dynamic" "k8s.io/kubernetes/test/e2e/framework" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) var _ = SIGDescribe("CustomResourceDefinition Watch", func() { f := framework.NewDefaultFramework("crd-watch") Context("CustomResourceDefinition Watch", func() { /* Testname: crd-watch Description: Create a Custom Resource Definition and make sure watches observe events on create/delete. */ It("watch on custom resource definition objects", func() { framework.SkipUnlessServerVersionGTE(crdVersion, f.ClientSet.Discovery()) const ( watchCRNameA = "name1" watchCRNameB = "name2" ) config, err := framework.LoadConfig() if err != nil { framework.Failf("failed to load config: %v", err) } apiExtensionClient, err := clientset.NewForConfig(config) if err != nil { framework.Failf("failed to initialize apiExtensionClient: %v", err) } noxuDefinition := testserver.NewNoxuCustomResourceDefinition(apiextensionsv1beta1.ClusterScoped) noxuVersionClient, err := testserver.CreateNewCustomResourceDefinition(noxuDefinition, apiExtensionClient, f.ClientPool) if err != nil { framework.Failf("failed to create CustomResourceDefinition: %v", err) } defer func() { err = testserver.DeleteCustomResourceDefinition(noxuDefinition, apiExtensionClient) if err != nil { framework.Failf("failed to delete CustomResourceDefinition: %v", err) } }() ns := "" noxuResourceClient := newNamespacedCustomResourceClient(ns, noxuVersionClient, noxuDefinition) watchA, err := watchCRWithName(noxuResourceClient, watchCRNameA) Expect(err).NotTo(HaveOccurred()) watchB, err := watchCRWithName(noxuResourceClient, watchCRNameB) Expect(err).NotTo(HaveOccurred()) testCrA := testserver.NewNoxuInstance(ns, watchCRNameA) testCrB := testserver.NewNoxuInstance(ns, watchCRNameB) By("Creating first CR ") testCrA, err = instantiateCustomResource(testCrA, noxuResourceClient, noxuDefinition) Expect(err).NotTo(HaveOccurred()) expectEvent(watchA, watch.Added, testCrA) expectNoEvent(watchB, watch.Added, testCrA) By("Creating second CR") testCrB, err = instantiateCustomResource(testCrB, noxuResourceClient, noxuDefinition) Expect(err).NotTo(HaveOccurred()) expectEvent(watchB, watch.Added, testCrB) expectNoEvent(watchA, watch.Added, testCrB) By("Deleting first CR") err = deleteCustomResource(noxuResourceClient, watchCRNameA) Expect(err).NotTo(HaveOccurred()) expectEvent(watchA, watch.Deleted, nil) expectNoEvent(watchB, watch.Deleted, nil) By("Deleting second CR") err = deleteCustomResource(noxuResourceClient, watchCRNameB) Expect(err).NotTo(HaveOccurred()) expectEvent(watchB, watch.Deleted, nil) expectNoEvent(watchA, watch.Deleted, nil) }) }) }) func watchCRWithName(crdResourceClient dynamic.ResourceInterface, name string) (watch.Interface, error) { return crdResourceClient.Watch( metav1.ListOptions{ FieldSelector:  "metadata.name=" + name, TimeoutSeconds: int64ptr(600), }, ) } func instantiateCustomResource(instanceToCreate *unstructured.Unstructured, client dynamic.ResourceInterface, definition *apiextensionsv1beta1.CustomResourceDefinition) (*unstructured.Unstructured, error) { createdInstance, err := client.Create(instanceToCreate) if err != nil { return nil, err } createdObjectMeta, err := meta.Accessor(createdInstance) if err != nil { return nil, err } // it should have a UUID if len(createdObjectMeta.GetUID()) == 0 { return nil, fmt.Errorf("missing uuid: %#v", createdInstance) } createdTypeMeta, err := meta.TypeAccessor(createdInstance) if err != nil { return nil, err } if e, a := definition.Spec.Group+"/"+definition.Spec.Version, createdTypeMeta.GetAPIVersion(); e != a { return nil, fmt.Errorf("expected %v, got %v", e, a) } if e, a := definition.Spec.Names.Kind, createdTypeMeta.GetKind(); e != a { return nil, fmt.Errorf("expected %v, got %v", e, a) } return createdInstance, nil } func deleteCustomResource(client dynamic.ResourceInterface, name string) error { return client.Delete(name, &metav1.DeleteOptions{}) } func newNamespacedCustomResourceClient(ns string, client dynamic.Interface, definition *apiextensionsv1beta1.CustomResourceDefinition) dynamic.ResourceInterface { return client.Resource(&metav1.APIResource{ Name:       definition.Spec.Names.Plural, Namespaced: definition.Spec.Scope == apiextensionsv1beta1.NamespaceScoped, }, ns) } </ins>

I am suspicious that CRD watch is not working, but was unable to find e2e test coverage to persuade myself that it is working. cf .
(in particular, I'm looking for the update tests.)
I think this was due to enabling protobuf, though it also failed in a curl for me, so something strange is happening. (Maybe we have to explicitly ask for json?)
We test CRD watch in and in , but not in . I agree we should add e2e tests as a requirement before GA, and also for the sake of defining conformance.
/area custom-resources
Are we talking about CRD or CR watch and protobuf? The later with protobuf obviously does not work, as commented.
I'm not sure "obviously" is correct - list works, for example. IIRC watch with protobuf just silently fails to do anything - it doesn't return an error. And yes I mean CR watch :-)
it should obviously not work with protobuf, but in the sense that the negotiating serializer switches to JSON if you offer it json and protobuf as alternatives. This seems to be broken.
/cc
I'd like to take this up. I'm familiarising myself with the apiextension code base, please give me some time to come up with a PR :)
We already have integration tests for this. Is an e2e test really needed?
I think we do need an e2e test for CRD watch (among other things) as a requirement for CRD going GA. The most pressing need is to define conformance (which only e2e tests can do). Currently we only test that you can create and delete a CRD, but we don't test that you can actually do anything with it: We should expand the conformance (e2e) test for CRD to include checking that the custom resource gets installed and is usable, including watches. If not for conformance, I think an e2e test for CRD watch would still be appropriate since it's an important end-user flow (a thing we expect users to do on purpose; not an edge case). For example, the apiextensions-apiserver integration test might miss problems that come from embedding into kube-apiserver. Similarly, the kubectl tests are meant to test kubectl client functionality, not to protect against regressions on the server side.

Someone might reasonably change those tests in a way that suits kubectl, without realizing that doing so could leave some server code untested at the e2e level. We also need the e2e test to make sure we honor the guarantees of the raw API, to ensure clients other than kubectl will work.
For our developer purpose I would say that e2e tests are not needed when integration tests are in-place. We can even test with a real kube-apiserver including embdded into kube-apiserver if we want that. With conformance tests might have a point. Though I wonder where the line is between e2e and features we test in integration or unit tests only. We don't test all behaviour in e2e today, by far.
Since this has the help-wanted label, adding some useful info for new contributors to get started: About CRDs: If you are new to the Kubernetes API in general, I'd recommend reading the first. CustomResourceDefinition (CRD) is a way to extend the Kubernetes API. You can read more about it here: If you'd like to do some additional reading about this, I had written something up during the time I did GSoC (shamless plug): About testing: There are lots of integration tests for CRDs and you can get a general idea about testing for CRDs and apiextensions-apiserver in general. There is for CRDs. You can use this and other e2e tests to understand how to write them. contains useful information about how you can run tests. gives general info about e2e tests and mentions tests about writing good e2e tests. Also, you can ping me on Slack (I am there) if you have further questions or need help! :)


Question: return &api.Pod{}, nil } <ins> func (c *FakePods) Watch(label, field labels.Selector, resourceVersion string) (watch.Interface, error) { c.Fake.Actions = append(c.Fake.Actions, FakeAction{Action: "watch-pods", Value: resourceVersion}) return c.Fake.Watch, c.Fake.Err } </ins> func (c *FakePods) Bind(bind *api.Binding) error { c.Fake.Actions = append(c.Fake.Actions, FakeAction{Action: "bind-pod", Value: bind.Name}) return nil

PodInterface should have Watch method behaving in the same way as ServiceInterface.Watch does. Use case: a monitoring system (namely Prometheus) expects to get the list of monitored servers performing a DNS-SD query. Special service (similar to kube2sky shipped with Kubernetes) mirrors status of running Pods into etcd, which is in turn monitored by SkyDNS server and exported as DNS-SD. Schematically the data flow looks like this: Kubernetes -kube2sky -etcd -SkyDNS -Prometheus. The biggest challenge is to deliver list of all running pods from Kubernetes to kube2sky. Currently kube2sky can only poll Kubernetes with PodInterface.List() each N seconds, which is not efficient. Having a Watch method would be much better.
cc Do you know why we do not support watch?
Every object should support Watch.
Watching status doesn't currently work -- that's .
I agree with OP. This is just an accident of history; some boilerplate in the client isn't completely filled out. As Brian mentions, not every field is currently watchable but that's being addressed elsewhere.
This is relevant to your interests potentially w/ your work on
This is a pretty simple change if someone wants to get their feet wet. :)
I'll work on it.


Question: --can-ip-forward  --metadata-from-file "$3" "$4"; then if (( attempt > 5 )); then <del> echo -e "${color_red}Failed to create instance template $1 ${color_norm}" </del> <ins> echo -e "${color_red}Failed to create instance template $1 ${color_norm}" >&2 </ins> exit 2 fi <del> echo -e "${color_yellow}Attempt $(($attempt+1)) failed to create instance template $1. Retrying.${color_norm}" </del> <ins> echo -e "${color_yellow}Attempt $(($attempt+1)) failed to create instance template $1. Retrying.${color_norm}" >&2 </ins> attempt=$(($attempt+1)) else break

See issue as the roll-up for node upgrades. tl;dr: Update MIG template, then call a MIG rollingupdate. More details: When upgrading nodes, an in-place upgrade must worry about leftover state on the boot disk, VM configuration like IP tables, and the idempotency of salt. Getting a fresh VM instead would remove these concerns and provide a clean state for a new node. Furthermore, a versioned node spec () includes the tuple of (kubelet, kube-proxy, docker, kernel/OS). In the general case, changing docker or the kernel/OS means we have to stop the running pods. Once know we have to stop the pods, the simplest upgrade mechanism is to totally down the node and provision a new one. Provisioning a new node is also a great backup to have for node repairs. To be clear, though, this issue is entirely separate from node repairs; once we have the capability to grow the cluster by adding new nodes, this mechanism, and other repair mechanisms, will build on top of that capability. This was blocked by: nodes joining a running cluster (template-only metadata) References: The most relevant discussion regarding the worries of complications with state while doing in-place upgrades is .
FWIW, I think we need this as a fallback to repair completely borked nodes? That is, it may not be necessary for upgrades per se, but could end up being independently useful.
Thanks for the comment, I updated the description to be clear about what this issue is addressing. Having the ability to add new nodes to a cluster will both enable this upgrade path as well as give us a great "hammer" to use for repairs.
Working on this now.


Question: if err != nil || !existed || ip != ipPrime { t.Fatalf(`ensureStaticIP(%v, %v, %v, %v, %v) = %v, %v, %v; want %v, true, nil`, gce, ipName, serviceName, gce.region, ip, ipPrime, existed, err, ip) } <ins> // Ensure call with different name ipName = "another-name-for-static-ip" ipPrime, existed, err = ensureStaticIP(gce, ipName, serviceName, gce.region, ip, cloud.NetworkTierDefault) if err != nil || !existed || ip != ipPrime { t.Fatalf(`ensureStaticIP(%v, %v, %v, %v, %v) = %v, %v, %v; want %v, true, nil`, gce, ipName, serviceName, gce.region, ip, ipPrime, existed, err, ip) } </ins> } func TestEnsureStaticIPWithTier(t *testing.T) {

/kind bug What happened: If the cloud provider attempts to reserve an IP and it's already reserved, it should continue syncing. Instead, it's returning an error and causing the service controller to retry. The above code only checks for 409 status code, but we need to also check for code 400. If I recall correctly, 409 happens for naming conflicts while 400 occurs for external address conflict. How to reproduce it (as minimally and precisely as possible): a load balancer service address to static with a random name. an LB sync. /sig network /area platform/gce
I don't think we gain anything by backporting the func to 1.7, right? Checking for both status codes seems to be sufficient.
Correct. It could be a minor performance optimization, but wouldn't help with the correctness. Agreed.
Actually, it's not sufficient. Right after that block of code, it tries to retrieve the address by name - a name which may be the user's. I'm thinking this issue is not worth fixing. I don't want to make more changes for a minor edge case.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
/remove-lifecycle stale

We allow users to specify in the Service object. If the user later removes this from the Service Spec, the controller will fail to sync. For example: creates an LB for Service.Spec.LoadBalancerIP = "1.1.1.1" with a user-owned static IP address. updates the Service to remove the requested LoadBalancerIP tries to used by the existing forwarding rule (i.e., "1.1.1.1"). It encounters a conflict. Controller assumes it owns the IP and by name but fails. We can fix the controller to continue syncing and using the user-owned IP for the LB, or create a new address for the LB. The former seems less disruptive to me, but the downside is that if user will not be able to let other components use this IP (i.e., 1.1.1.1) unless they explicitly delete the Service. I think it's ok but would like more opinions. thoughts?
If a service is updated to remove a static address, then it seems like that would be a deliberate move on the part of the user. It seems like it should probably release the address and provision a dynamic one, even if this is disruptive. How do we handle the reverse? If a user has an already existing LoadBalancer service, and they add the after the fact? Would this release the dynamic address and recreate the forwarding rule with the reserved static address?
It seems like the creating a new address makes sense here... General comment: given the combinations (and the bugs we have been running into), I think this is a problem that all of the potential state transitions for the load balancer have not been fully spec'd out.
Creating a new address seems nice but the implementation is a nightmare. The service controller doesn't give you a previous Service spec, so you don't know what's changed (controller might not know either if spec was modified while offline). Given a desired service spec without a how do you decide to ditch your current IP and grab a new one? If we were to call to evaluate the situation, it's highly likely users will accidentally cause LBs to change IPs during typical usage. Imagine a user has a dynamically allocated IP for a service, but they want to promote it for safe keeping. The controller would think it needs to surrender the IP to the user's reservation.

The only correct process would be to set before promoting the address; however, that will surely be a missed point in docs and result in unhappy users. I lean towards the LB continuing use of the IP. Besides the easy option of deleting & re-creating the service, they could change the to another static address then demote the address. I'm not too familiar with the ideological boundaries of controllers and spec, so let me know if this crosses the line. What if the service controller set to the created IP on first sync if the IP wasn't pre-specified? The controller could rely on loadBalancerIP to always indicate which IP to use. If the field is empty (blanked by user), then it's a signal to get a new IP.
By spec, I mean we should write down the careful reasoning that you just wrote in a permanent place :-)
would it then make sense for to be immutable on an already created service?
+1 +1 Was thinking the same. Unlike the many annotations we support, updates to this actual field could be validated and blocked at the apiserver... Changing IP is disruptive anyway. Could we simply ask the user to re-create the Service?
Of course, this could be an API-breaking change, and other cloud providers may actually be supporting this properly.
I just ran across this today, but in a more likely (IMO) case. 1) create the Service, type=LoadBalancer 2) let the IP be auto-allocated 3) manually promote that IP to static and give it a meaningful name 4) observe errors from service controller trying to allocate that IP Since my Service does not spec "loadBalancerIP", the question is, do we: a) use the IP, but leave it as static b) use the IP, but demote it back to ephemeral c) allocate and use a new IP C is completely hostile. B is subtly hostile. A seems correct. Can it be implemented? If we find that the IP exists but the name of it is not what we expect, we should treat it as user-provided and leave it alone. This is still nasty because it logs a failure in the GCP activities log for every loop, but I don't know how to avoid that without getting a list of IPs and scanning for the IP we want.
Filed for the looping.

+1 as long as we can rely on the "naming" of the address to decide whether the controller owns the address and can promote/demote it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still relevant
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still relevant
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
any update on this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


Question: } } <ins> func TestNodePortUDP(t *testing.T) { nodeIP := net.ParseIP("100.101.102.103") ipt := iptablestest.NewFake() ipvs := ipvstest.NewFake() ipset := ipsettest.NewFake(testIPSetVersion) fp := NewFakeProxier(ipt, ipvs, ipset, []net.IP{nodeIP}) svcIP := "10.20.30.41" svcPort := 80 svcNodePort := 3001 svcPortName := proxy.ServicePortName{ NamespacedName: makeNSN("ns1", "svc1"), Port:           "p80", } makeServiceMap(fp, makeTestService(svcPortName.Namespace, svcPortName.Name, func(svc *api.Service) { svc.Spec.Type = "NodePort" svc.Spec.ClusterIP = svcIP svc.Spec.Ports = []api.ServicePort{{ Name:     svcPortName.Port, Port:     int32(svcPort), Protocol: api.ProtocolUDP, NodePort: int32(svcNodePort), }} }), ) epIP := "10.180.0.1" makeEndpointsMap(fp, makeTestEndpoints(svcPortName.Namespace, svcPortName.Name, func(ept *api.Endpoints) { ept.Subsets = []api.EndpointSubset{{ Addresses: []api.EndpointAddress{{ IP: epIP, }}, Ports: []api.EndpointPort{{ Name: svcPortName.Port, Port: int32(svcPort), }}, }} }), ) fp.nodePortAddresses = []string{"0.0.0.0/0"} fp.syncProxyRules() // Check ipvs service and destinations epVS := &netlinktest.ExpectedVirtualServer{ VSNum: 2, IP: nodeIP.String(), Port: uint16(svcNodePort), Protocol: string(api.ProtocolTCP), RS: []netlinktest.ExpectedRealServer{{ IP: epIP, Port: uint16(svcPort), }}} checkIPVS(t, fp, epVS) // check ipSet rules epIPSet := netlinktest.ExpectedIPSet{ kubeNodePortSetUDP: {{ Port:     svcNodePort, Protocol: strings.ToLower(string(api.ProtocolUDP)), SetType:  utilipset.BitmapPort, }}, } checkIPSet(t, fp, epIPSet) // Check iptables chain and rules epIpt := netlinktest.ExpectedIptablesChain{ string(kubeServicesChain): {{ JumpChain: string(KubeNodePortChain), MatchSet: kubeNodePortSetUDP, }}, string(KubeNodePortChain): {{ JumpChain: string(KubeMarkMasqChain), MatchSet: "", }}, } checkIptables(t, ipt, epIpt) } </ins> func TestNodePort(t *testing.T) { ipt := iptablestest.NewFake() ipvs := ipvstest.NewFake()

Is this a BUG REPORT or FEATURE REQUEST? /kind bug What happened: In ipvs proxy mode, most service functions like "nodeport/loadbalance" is realized by the association of iptables/ipset/ipvs, yet most ipvs proxier ut only verified ipvs rules, which can't ensure service is functional. What you expected to happen: Check both ipvs rules and iptables chains/rules, ipset entries in ipvsproxier ut.
/sig network /area ipvs /assign
What's the current UT coverage number?
Current is 50.2%.
So, can we achieve 70%?
Yeah, we can do that, and that's definitely necessary.


Question: func TestProxyHandler(t *testing.T) { target := &targetHTTPHandler{} <del> targetServer := httptest.NewTLSServer(target) </del> <ins> targetServer := httptest.NewUnstartedServer(target) if cert, err := tls.X509KeyPair(svcCrt, svcKey); err != nil { t.Fatal(err) } else { targetServer.TLS = &tls.Config{Certificates: []tls.Certificate{cert}} } targetServer.StartTLS() </ins> defer targetServer.Close() tests := map[string]struct {

an APIService that routes to an external API server (e.g. service catalog) via a service in namespace a caBundle in the APIService the external API server with a certificate signed for Result: Normal API requests (get/put/post) work as expected Normal watch requests work as expected Upgrade requests (e.g. websocket watch requests) return a 500 error with text like cc


Question: e2eskipper.SkipUnlessFeatureGateEnabled(features.ImageVolume) }) <del> createPod := func(ctx context.Context, volumes []v1.Volume, volumeMounts []v1.VolumeMount) { var selinuxOptions *v1.SELinuxOptions if selinux.GetEnabled() { selinuxOptions = &v1.SELinuxOptions{ User:  defaultSELinuxUser, Role:  defaultSELinuxRole, Type:  defaultSELinuxType, Level: defaultSELinuxLevel, } ginkgo.By(fmt.Sprintf("Using SELinux on pod: %v", selinuxOptions)) } </del> <ins> createPod := func(ctx context.Context, podName, nodeName string, volumes []v1.Volume, volumeMounts []v1.VolumeMount, selinuxOptions *v1.SELinuxOptions) { </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      podName, Namespace: f.Namespace.Name, }, Spec: v1.PodSpec{ <ins> NodeName:      nodeName, </ins> RestartPolicy: v1.RestartPolicyAlways, SecurityContext: &v1.PodSecurityContext{ SELinuxOptions: selinuxOptions,

Related to and SIG storage meeting notes: From me: It is related to image volume plugin and selinux. Is it a known limitation of all the volume plugins when using selinux? I have a different opinion. If two pods don't share the same mountpoint (hostpath), they can run the same node when the selinux is enabled. the current implementation of cri-o is sharing the same hostpath when two pods use the same image volume. More details about my idea: From Sharing each volume between namespaces is kind of natural. Sharing PVs not possible but sharing image quite common. From Although for most volume plugins it is true that, they can't be mounted with 2 different selinux context on same node, we want to understand why is this not possible for Imagevolume source. For a traditional storage persistent storage plugins - this limitation is underlying limitation of storage backend/Linux kernel etc. For ImageVolume source, it seems like a simple solution which unpacks image to two different hostpath would work and not run into this limitation. can you specify, why this limitation exists for ImageVolume plugins? From we choose to keep it simple and follow the lead of other volume types. supporting multiple selinux types would mean making more complicated the way we mount the image into the container AFAIR. But agreed we can chat about this elsewhere, in the meantime for the purposes of this PR: this is consistent with behavior of other volume plugins and is expected behavior. Let's discuss the SELinux behavior for image volumes in this issue. cc /sig node /sig storage
In my opinion, from an end-user perspective, the image volume may act like a read-only with a pre-populated content via image. Two pods from different namespaces can share the same image and run on the same node.
Pasted from According to , the containerd doesn't have the same limitation.
Given that OCI/Image Volume Source support is in alpha, I'd prefer that we consider discussing this as part of the original feature. We can discuss this here but I don't know if we need a separate KEP for this yet.

Yes, so far, in our conversations at the containerd community meetings we decided to start this feature out using a single pod scope for the pod scoped image volume host path because in containerd each runtime handler may have a different snapshotter (possibly on a different fs), or at least that's the plan. Additionally, the pods may have different os versions on windows indicating a different platform can be selected on the same node and this sort of versioning may extend beyond to other os providers. We could ask Kirtana to confirm the versioning issue and status of platform version selection over the platform index manifest, if the snapshotter reason isn't enough to at least start with per pod scope. If this is insufficient support we can certainly discuss. Cheers, Mike
No need a separate KEP. We can update the original KEP once this issue is resolved.
I'm working on a technical solution (just a PoC) in , not sure how feasible that is as time of writing, though.
yeah I agree I don't think we need a different feature for this, especially because it's mostly an implementation detail in the runtime. would you open this as an issue in kubernetes/kubernetes instead of the enhancements repo, or even better in the cri-o repo?
/transfer kubernetes
I moved this issue into kubernetes repo.
/triage accepted /priority important-soon
thanks!
Copid from kep: As merged, do we need to change the KEPs to reflect the new behavior?
no, nothing needs to be changed from a Kubernetes any more :+1:
/remove-sig storage


Question: http_archive( name = "io_k8s_repo_infra", <del> sha256 = "462dcc769f29e9eee16499edb818e2fc2ffc24efea28300d53bbc098d27d322c", strip_prefix = "repo-infra-f85734f673056977d8ba04b0386394b684ca2acb", urls = mirror("https://github.com/kubernetes/repo-infra/archive/f85734f673056977d8ba04b0386394b684ca2acb.tar.gz"), </del> <ins> sha256 = "f6d65480241ec0fd7a0d01f432938b97d7395aeb8eefbe859bb877c9b4eafa56", strip_prefix = "repo-infra-9f4571ad7242bf3ec4b47365062498c2528f9a5f", urls = mirror("https://github.com/kubernetes/repo-infra/archive/9f4571ad7242bf3ec4b47365062498c2528f9a5f.tar.gz"), </ins> ) http_archive(

What happened: that uses Ubuntu image fails during Kubernetes installation: Full log: Indeed, is not present on nodes. Only and subdirectories are present in . I don't know what code removed (or did not package) and symlinks, but they should be there. What you expected to happen: Kubernetes is installed correctly and e2e tests start. How to reproduce it (as minimally and precisely as possible): Run in a PR. Anything else we need to know?: Prow job: It seems to be the only one that uses Ubuntu image. This command works and installs Kubernetes correctly: is present there. Dunno what the prow job does differently.
/sig testing
the nodes receive files staged via the release scripts and downloaded by the cluster/ scripts, I don't think sig testing will be the right group to fix this. SIG node has some owners related to the various images that might have some idea how this is setup
/sig release this is probably an issue in the release scripts
cc:
PTAL, it looks like a stupid bug somewhere.
Q: Did this ever run successfully?
Most probably not, I haven't found a working e2e tests on Ubuntu. Still, works well, there must be something slightly different in test setup scripts.
ping again? Short recap: This works: is there. ( is symlink to ) This does not work: Fails with: does not exist. The prow job must do something slightly different than kube-
The serial job is using GCI (COS): Is the non-serial job working? It uses ubuntu: If not, my guess from the last time I remember looking was checking the release bundle and ensuring it includes the ubuntu dir
No, same error. created by contains , released bundles (e.g. ) contains it too. Otherwise would not work with an Ubuntu image. So what does the prow job differently? Does something there prune the scripts? is just a symlink to , does it make a difference?
Found it, creates without any symlinks. It looks like bazel limitation. I filled to explicitly create the symlinks when tarball is created. BTW, if and use different way to produce tarballs, who / when tests the ones created by ?
Most of CI is using make release or equivalent. Presumbit uses bazel. Jan Šafránek <:
/area release-eng /milestone v1.16 /priority important
/priority important-soon


Question: func IsDeleted(info *resource.Info, o *WaitOptions) (runtime.Object, bool, error) { endTime := time.Now().Add(o.Timeout) for { <ins> if len(info.Name) == 0 { return info.Object, false, fmt.Errorf("resource name must be provided") } </ins> gottenObj, err := o.DynamicClient.Resource(info.Mapping.Resource).Namespace(info.Namespace).Get(info.Name, metav1.GetOptions{}) if apierrors.IsNotFound(err) { return info.Object, true, nil

Is this a BUG REPORT or FEATURE REQUEST?: What happened: I created a job with metadata.generateName twice and waited for completion, but got: panic: interface conversion: runtime.Object is unstructured.UnstructuredList, not unstructured.Unstructured What you expected to happen: kubectl to wait for the last job to complete. How to reproduce it (as minimally and precisely as possible): kubectl create -f kubectl create -f kubectl wait --for=condition=complete -f (shortened): apiVersion: batch/v1 kind: Job metadata: generateName: my-job- Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I have to choose a sig, but I am not sure which one is right...
Reiterating the mentions to trigger a notification: <detailsIn response to : I have to choose a sig, but I am not sure which one is right... Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli /sig apps
/assign
kubectl get -f also doesn't work as expected.
/remove-sig apps


Question: // schedulingTimeout is longer specifically because sometimes we need to wait // awhile to guarantee that we've been patient waiting for something ordinary // to happen: a pod to get scheduled and move into Ready <del> const schedulingTimeout = 5 * time.Minute </del> <ins> const schedulingTimeout = 10 * time.Minute </ins> var _ = framework.KubeDescribe("DisruptionController", func() { f := framework.NewDefaultFramework("disruption")

https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite}
https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite} Happened on a presubmit run in .

https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite}
https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite}
https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] DisruptionController evictions: too few pods, replicaSet, percentage =should not allow an eviction {Kubernetes e2e suite} Happened on a presubmit run in .


Question: } baseNamespace := baseNamespaceObj.(*api_v1.Namespace) if baseNamespace.DeletionTimestamp != nil { <del> nc.delete(baseNamespace) </del> <ins> if err := nc.delete(baseNamespace); err != nil { glog.Errorf("Failed to delete %s: %v", namespace, err) nc.eventRecorder.Eventf(baseNamespace, api.EventTypeNormal, "DeleteFailed", "Namespace delete failed: %v", err) nc.deliverNamespace(namespace, 0, true) } </ins> return }

In PR the specific resource types are handled explicitly. Need to delete them in more generic way, in a similar way that namespace controller does.
- switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)
did fix this?
Please close or move to next-candidate with work remaining to be done.
Yes fixed by


Question: <ins> # ensure go's non-cgo resolver respects /etc/hosts hosts: files dns </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: My broke after , after much debugging we discovered that the API server could not reach itself on , further debugging showed that DNS lookups for localhost were occuring. Long story short, while we have entries for localhost, the api server was not respecting them. What you expected to happen: our standard control plane deployments should not lookup localhost via dns. they should respect /etc/hosts How to reproduce it (as minimally and precisely as possible): create a cluster and monitor network traffic. then try again after adding a minimal to the api server pod. Anything else we need to know?: Prior art: This happens because we build with go's build in DNS resolution, which respects glibc's , but we build images based on busybox for some components (like the apiserver) which does not ship with one. Environment: kubernetes using Kubernetes version (use ): /sig release
/assign cc Thanks a million to and for looking into this with me! this was tricky to resolve.
We should be able to fix this by just adding a minimal to the images make go's resolver happy. alternatively we can standardize on eg which already ships one
oh wow. tough one to debug :)
/milestone v1.13 /priority important-soon
I have a PR under way, will file after lunch!
For now adding the , but discussing with and it sounds like switching to debian-base everywhere is the best long term answer. It will actually save a mb or so on each node :sweat_smile: ... and give a more consistent base for component images to build on.
I just got bit by this.
This issue will bite anyone deploying a statically linked go binary, not just the api server. Can we get kubelet to slide in a default that is sane so it wont keep biting people? Similar to
We'd need a feature request to sig-node and probably sig-network. I do like this idea, but, is used for things other than networking so injecting one might be slightly problematic. In this case it was fine because we control the images and the binaries.

People deploying go images need to make sure they have a sane for now, or not deploy staticly linked binaries, unfortunately.
For 1.13, we should just document the problem
would it be difficult to add the file only if it doesn't exist in the image?
workaround for 1.12: echo hosts: files dns myhostname and add to the ClusterConfig:
Yeah that work around for now, I'm not sure how difficult / expensive it would be for the kubelet to conditionally add this, and it might be a surprising behavior change for arbitrary containers / pods.
it may be surprising, but its very surprising now too, that dns takes precident to hosts file unexpectedly. :( I'd rather it be secure by default and surprise them that its safe. I didn't even know glibc had this backwards behavior when was missing. I haven't run up against a glibc based system that didn't have an nsswitch file. All distro's add one.
hmm... etcd is crashing too.... I hope there is an equiv of etcdExtraVolumes....
never mind on the etcd thing that was something else. The above yaml tweaks did solve the issue.
We have just wasted a TON of time on this when a misconfigured Samsung TV on the other side of the world, plus lack of rigor or our DDNS policy broke our Kube 1.12 clusters. I really don't like that these control plane images are hitting DNS for every lookup of localhost. I am very keen to see implemented.
The control plane images in future releases already contain , which could be cherry picked back. Doing this in an automated fashion like that issue would require bringing a feature to sig-network and sig-node for discussion. I suspect supplying a file only if it doesn't exist in the image would be somewhat expensive compared to what we do currently, and it's definitely a breaking change. In the meantime, if you're containerizing go and not using CGO for networking, you need to add this to your own images. On Tue, Nov 20, 2018, 10:00 nickperry wrote:
I would argue, that 1.12 was a broken release. 1.11 and below did not have the issue.

As K8S is maturing, some of us, especially in more enterprise like environments, are not upgrading as fast, so the issues are being found slower. I think backporting the fix for the control plane images is appropriate to resolve the regression until a more universal fix can be put in place.
that sounds like a great WHODUNIT war story :)
generally I'm not upgrading production workloads particularly quickly either :-) We can file cherry picks back to 1.12, but lack of an in the control plane images is not new: EG does not contain an . IMO, setting on control plane pods was certainly broken, which I questioned, but not just for this reason, and that change was eventually reverted IIRC. /cc Regarding -- because Kubernetes wants to be stable there are processes for proposing and vetting features prior to adding them, particularly for user facing changes! If you'd like to see this, please discuss it with sig-network and sig-node. I don't personally have the bandwidth for this right now and I'm not even sure that it is even the right solution.
yes the change was reverted. it also spawned a little bit of a discussion here for a rather complicated workaround: given this affects images that are already deployed at GCR, it's unlikely that it can be cherry picked. the images do support revisions, but this would also mean porting changes to all the bundled tools that use them as hardcoded tags. also this means revisions for all patch release images. overall i see this as something that will not be approved by sig-network and the GCR maintainers.
I guess to prevent a future regression, we probably need a test that inserts a <some other addressrecord somewhere during the test.


Question: healthzServer:         healthzServer, ipvs:                  ipvs, ipvsScheduler:         scheduler, <del> ipGetter:              &realIPGetter{nl: NewNetLinkHandle()}, </del> <ins> ipGetter:              &realIPGetter{nl: NewNetLinkHandle(isIPv6)}, </ins> iptablesData:          bytes.NewBuffer(nil), filterChainsData:      bytes.NewBuffer(nil), natChains:             bytes.NewBuffer(nil), natRules:              bytes.NewBuffer(nil), filterChains:          bytes.NewBuffer(nil), filterRules:           bytes.NewBuffer(nil), <del> netlinkHandle:         NewNetLinkHandle(), </del> <ins> netlinkHandle:         NewNetLinkHandle(isIPv6), </ins> ipset:                 ipset, nodePortAddresses:     nodePortAddresses, networkInterfacer:     utilproxy.RealNetwork{},

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area ipv6 /area ipvs /sig network What happened: In an ipv6-only cluster with proxy-mode=ipvs the entries for NodePort in ipvs are created for ipv4 addresses. The target's (if there are any) consists of ipv6 addresses (wrongly) interpreted as ipv4; Attempts to use the NodePort with ipv6 addresses are black-holed. What you expected to happen: Ipv6 addresses to the nodes should be used for NotePort and the LB-targets shall be the ipv6 address of the pods. Traffic to NodePorts shall be distributed among the (ipv6) endpoints. How to reproduce it (as minimally and precisely as possible): Setup an ipv6-only cluster Create a service with type with endpoints Try to connect to the NodePort Do Anything else we need to know?: The nodes have both ipv4 and ipv6 addresses and it is possible that if the nodes only had ipv6 addresses it would work (I havn't tested). But even in an ipv6-only k8s cluster the nodes almost always need ipv4 addresses for other purposes. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm VMs on Ubuntu 18.04.01 LTS OS (e.g. from /etc/os-release): Own BusyBox based system Kernel (e.g. ):Install tools: None Others:
The NodeIPs() function seem to return only ipv4 addresses.
I can't see why (yet) since is used in the GetLocalAddresses function
/cc Please give a hint before I dive into the netlink code.
This issue is used as an example here;
The functions filters on Src != nil . But ipv6 addresses seems to have Src == nil always. As it is this function will never return any ipv6 addresses.


Question: func TestGenerateAPIPodStatusWithSortedContainers(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) <ins> testKubelet.fakeCadvisor.On("VersionInfo").Return(&cadvisorapi.VersionInfo{}, nil) testKubelet.fakeCadvisor.On("MachineInfo").Return(&cadvisorapi.MachineInfo{}, nil) testKubelet.fakeCadvisor.On("ImagesFsInfo").Return(cadvisorapiv2.FsInfo{}, nil) testKubelet.fakeCadvisor.On("RootFsInfo").Return(cadvisorapiv2.FsInfo{}, nil) </ins> kubelet := testKubelet.kubelet numContainers := 10 expectedOrder := []string{}

Currently we query the apiserver for the node (indirectly: ) before populating hostip on a pod. I believe this is because the node controller is responsible for calling into the cloud provider and populating the ip address on a node: This is disappointing for 2 reasons: makes using ips to identify nodes harder (). 't separate concerns sufficiently. If the reflector fails to watch nodes reliably pods won't get a hostip.
Also see .
I would argue that if we're unable to watch nodes reliably, we have many other problems. That seems like something we need to fix, rather than work around.
watch is watch, a separation of concerns between the kubelet (which is running on the node that has the hostip) and the apiserver is a different problem imo.
After discussion, we decided to just remove host IP from PodStatus. (It will still be available from "kubectl get nodes" if someone needs it.) does that sound OK with you?
I'm working on making the nodes register themselves rather than having the node controller do it for them (see which should turn into a PR later this week). Will that assuage your concerns here?
I hadn't looked at your PR. We discussed having Kubelet pull ths information from the cloud provider but thought it might not be a good idea due to (1) requiring Kubelet to understand the details of all the cloud providers, and (2) making very hard to rate limit the calls into the cloud provider. I could imagine an approach where we still use NodeController to get the node address, but the node still initiates the registration into the cluster. Not sure whether that's a good idea, though.
Kubelet won't necessarily (and really shouldn't) have the credentials necessary to contact the cloud provider. I don't remember why HostIP was to PodStatus, and I would favor removing it, except that it would be a breaking change. You could ask on google-containers@ whether anyone is relying upon it. HostIP is tricky, as hosts may have multiple addresses, in general, which is why NodeAddresses is now an array.
On GCE / AWS do you even need credentials to determine your external IP? I thought you got it from the metadata server running locally without needing to hit an Auth'd API endpoint.
HostIP wasn't the external IP AFAIK.

It's not well defined what it is, which is one reason it should be eliminated. Yes, one should be able to get the host IP from metadata, but the cloudprovider library doesn't currently do that.
I'm also in favor of removing HostIP from pods. But I don't agree with the reason in the OP (that sometimes it'll be blank when kubelet hasn't heard from apiserver)-- I think that's par for the course.
Should we schedule a meeting to discuss whether it's reasonable for Kubelets to pull their IPs directly from the cloud provider? What is the alternative -- a new API endpoint on the master where unregistered nodes can query their IP address from NodeController? Do nodes really need their IP address when they register with the master?
That sounds like a good idea (I think we can likely clear this up with a quick meeting rather than going back and forth for days in github comments).
We all agreed that we should remove HostIP from pods. Nodes do not need to provide their address(es) when they register. It's not part of the spec. They could provide whatever hostnames and/or addresses they can get through normal Linux mechanisms and/or could contact a local metadata server, but I don't think that's necessary, since the node controller will update the addresses once the node is registered, unless we want to support scenarios with no cloud provider and no external node database. In any case, we shouldn't discuss self-registration in this issue. There are other issues that cover that topic.
OK, so this issue just boils down to removing HostIP from PodStatus, and we'll discuss the rest in ?
Ping on this. e2e "Pods should get a host IP [Conformance]" is currently disabled everywhere. Bumping this to and .
Today Kubelet fills PodStatus.HostIP in generatePodStatus() in Master (e.g. NodeController) is not involved. Currently the test waits two minutes for a pod to get a HostIP Considering that we have proposed to remove this field completely, I think we should just bump up the timeout a bit and see if that fixes the test. Any other thoughts? I have also sent out email to the appropriate mailing lists to see if anyone is using PodStatus.HostIP, since we would like to get rid of it, as discussed upthread.

cc/
BTW the title of this issue says "Kubelet shouldn't require apiserver to populate host ip of pods." But it wasn't clear to me where the dependency on the API server is (unfortunately 's original comment points to a very old revision to so I'm not sure which line he was referring to). The work of finding the HostIP seems to be done by Kubelet.GetHostIP() / Kubelet.GetNode() which assumes Kubelet.nodeInfo is filled in -- is that the dependency? I couldn't find where that gets filled in. would be great if you could shed some light.
I haven't looked at this code recently but nodinfo appears to still be retrieved from a cache populated via apiserver watch:
Ah yes, that looks right, thanks (well, "right" in the sense of "that's where it gets populated," not in the sense of "it makes sense that kubelet is using code defined in the scheduler predicates for this"...) while you're here, do you have a suggestion of the best course of action? Long-term we want to get rid of PodStatus.HostIP, but short-term we need to do something about this test. How can we make populating the cache more reliable? Or should we just increase the timeout on the test? Or something else?
It looks like the kubelet already has the node address: , so assuming no one else is going to change the ip, we could just store it in the kubelet and add it for every pod? Unsure if this is a safe assumption.
I also haven't debugged this new set of test failures, so I'm not sure if it's failing for the same reason
... to make on all clouds
There isn't a new set of failures - Ike just noticed that the test is disabled, from when this issue was originally opened. could one of you comment on whether Prashanth's suggestion sounds reasonable?
I didn't parse through the entire issue since a lot of information is stale. Yes David is right, we moved to the logic of filling hostIP of Pod from apiserver to kubelet long time ago. Recording to the comment of Prashanth: Some users might have node with multiple network interfaces, and want to hostIP with specific one by updating Node object.
Fwiw I think we should turn the test on and handle things as they come up.

I think the major source of flake was handled when we did:
With respect to the long-term future of HostIP: I'm in favor of documenting the field as deprecated and asking users to contact us with their use case if they need it (in the API field documentation). As long as the field exists, I agree we should test that it's set correctly.
in this comment you wrote "Ping on this. e2e "Pods should get a host IP [Conformance]" is currently disabled everywhere. Bumping this to P1 and flake." What made you say it is disabled? I looked at Jenkins and it seems to be running it: (root)/Kubernetes%%20suite/PodsshouldgetahostIPConformance/history/ I looked in and didn't see any special handling of [Conformance]. There is an env var CONFORMANCETESTSKIP_REGEX in hack/gingko- but it doesn't seem to be set anywhere. So I don't see anything that should be causing this test to not be run. (As always there's a high likelihood I'm misunderstanding something.)
Sorry, I'm totally blind, the Jenkins history I linked to clearly shows it is being skipped. But I'm still not sure what is causing it to be skipped.
Sorry I wasn't clear; it's being skipped because it was marked pending via . Let's agree to use and labels in the future for clarity's sake.
Also see for use cases of "I need to know my local node" and possible fixes.
I'm going to remove the flake label, since has been merged and that was the suggestion here We can re-add the label if it flakes again. I'm not closing the issue, because we want to deprecate the field longer-term.


Question: f.PodClient().DeleteSync(burstablePodName, &metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) f.PodClient().DeleteSync(bestEffortPodName, &metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) f.PodClientNS(kubeapi.NamespaceSystem).DeleteSync(criticalPodName, &metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) <del> err := f.ClientSet.SchedulingV1().PriorityClasses().Delete(systemCriticalPriorityName, &metav1.DeleteOptions{}) framework.ExpectNoError(err) </del> // Log Events logPodEvents(f) logNodeEvents(f)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Setting PodPriority directly when adminssion Priority enable,the Pod is rejected. To fix this, we need to create new priority classes for these tests. Which jobs are failing: Which test(s) are failing: I0822 19:01:41.165] should be able to create and delete a critical pod [It] Since when has it been failing: Testgrid link: Reason for failure: because adminssion Priority enableIf the priority class is not found, the Pod is rejected. Anything else we need to know:
/area e2e-test-framework /sig testing /kind bug /priority important-soon
This has now changed to this error: /reopen
Reopened this issue. <detailsIn response to : This has now changed to this error: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For reference: testgrid link /milestone v1.17
is this still in progress?
pr has updated, PTAL.
looks like has to merge for this to close
/milestone v1.18


Question: http.NotFound(w, r) } } <ins> // GenerateExpectedCgroupParent returns cgroup parent in syntax expected by cgroup driver func (ds *dockerService) GenerateExpectedCgroupParent(cgroupParent string) (string, error) { if len(cgroupParent) > 0 { // if docker uses the systemd cgroup driver, it expects *.slice style names for cgroup parent. // if we configured kubelet to use --cgroup-driver=cgroupfs, and docker is configured to use systemd driver // docker will fail to launch the container because the name we provide will not be a valid slice. // this is a very good thing. if ds.cgroupDriver == "systemd" { systemdCgroupParent, err := kubecm.ConvertCgroupFsNameToSystemd(cgroupParent) if err != nil { return "", err } cgroupParent = systemdCgroupParent } } glog.V(3).Infof("Setting cgroup parent to: %q", cgroupParent) return cgroupParent, nil } </ins>

Kubernetes version (use ): v1.5.0-alpha.1.2189+-dirty docker: 1.12.3 Environment: fedora24 docker-engine-1.12.3-1.fc24.x86_64 What happened: if docker Cgroup Driver=cgroupfs, and set --cgroup-driver=systemd in kubelet, then enable --experimental-cgroups-per-qos=true; The container cgroup some is not correct. How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: 1) docker Cgroup Driver=cgroupfs and --cgroup-driver=cgroupfs in kubelet works well 2) docker Cgroup Driver=systemd and --cgroup-driver=systemd in kubelet works well
help check, thanks.
In Kubernetes 1.6, we can work to detect this case and return an error. IMHO its a configuration error to have docker running with a cgroupfs driver paired with the kubelet running the systemd cgroup driver. /cc
We can also document the compatibility matrix for now.


Question: {utiliptables.TableFilter, kubeProxyFirewallChain}, {utiliptables.TableFilter, kubeSourceRangesFirewallChain}, {utiliptables.TableFilter, kubeIPVSFilterChain}, <ins> {utiliptables.TableFilter, kubeIPVSOutFilterChain}, </ins> } // ipsetInfo is all ipset we needed in ipvs proxier

When using loadBalancerSourceRanges to allow access only from CIDRs outside the cluster segment, access from the K8s nodes (or pods with hostNetwork:true) should be rejected, but for proxy-mode=ipvs it works. /sig network /area kube-proxy /area ipvs Access from K8s nodes should be rejected in the same way as for proxy-mode=iptables Define a service with loadBalancerSourceRanges that allows access from a cidr outside the cluster, e.g. The K8s nodes have addresses on 192.168.1.0/24 and fd00::c0a8:100/120. Make sure a loadBalancerIP is assigned (in this example 10.0.0.0/32) and try to access it from a K8s node: Same for both ipv4 and ipv6. This is related to since a fix requires that the passed to the is correct. Please see v1.27.4 and master None Linux 6.4.4 None N/A N/A
wanted to work on this
Great! But please coordinate with I hope that one get accepted, and then ~the s (both families) passed to the proxiers should be usable.~ No, they are not, please see
Discussed this one in the SIG Network meeting today: we agree this is something we'd like to implement, however we weren't sure exactly how to do it with ipvs during the meeting. Whoever takes this on needs to come up with and communicate a clear plan of how it will be accomplished. /triage accepted
My comment is not feasible. For proxy-mode=ipvs the src and dst is both loadBalancerIP. So the nodeIP will not help :disappointed:
I did a small experimented with kind + metallb. I created a loadbalancer service backed by . loadBalancerIP: 172.18.255.200 podIP: 10.244.1.3 conntrack picks the src and dst both as loadBalancerIP. Tried to create an iptables rules matching source and destination with loadbalancerIP to drop packets but it didn't work. After some exploration I found and with the following rules I was able to prevent access to lbIP from k8s nodes / host network.

The solution looks hacky though, we only need this rule if source ranges are defined for the service and node ip is not covered by any of them. Will try to program this in the ipvs proxier.
Good find! :+1: I had missed the ipvs extension. I started to check nodeIP and the (flawed) option to experiment with setting a nodeIP as "src" for some routes to dodge the lbip-both-as-src-and-dst problem, and an old "vision" I have to not assign clusterIPs to at all, but have one rule to replace them all, like: It can be done, and in a cluster with 1000's of clusterIP's it would simplify


Question: // A pod is besteffort if none of its containers have specified any requests or limits. // A pod is guaranteed only when requests and limits are specified for all the containers and they are equal. // A pod is burstable if limits and requests do not match across all containers. <del> func GetPodQOS(pod *corev1.Pod) corev1.PodQOSClass { requests := corev1.ResourceList{} limits := corev1.ResourceList{} </del> <ins> func GetPodQOS(pod *core.Pod) core.PodQOSClass { requests := core.ResourceList{} limits := core.ResourceList{} </ins> zeroQuantity := resource.MustParse("0") isGuaranteed := true <del> for _, container := range pod.Spec.Containers { </del> <ins> allContainers := []core.Container{} allContainers = append(allContainers, pod.Spec.Containers...) allContainers = append(allContainers, pod.Spec.InitContainers...) for _, container := range allContainers { </ins> // process requests for name, quantity := range container.Resources.Requests { if !isSupportedQoSComputeResource(name) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In kubectl library, logic is not the same as the one in What you expected to happen: logic should be the same. How to reproduce it (as minimally and precisely as possible): Create a pod with a guaranteed QoS class and a best-effort QoS initcontainer. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
It seems to be duplicate functions here.
The intention for duplicates is coming from the fact that kubectl can't rely on packages from kuberentes main repository. Either way they should both be identical except for the fact that the internval version will use internal APIs, whereas kubectl's version will use versioned.

feel free to open a PR updating kubectl version updating it to match both versions.
/triage accepted /assign
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details