Question: } return nodeutil.GetNodeHostIP(node) } <ins> // getHostIPAnyway attempts to return the host IP from kubelet's nodeInfo, or the initialNodeStatus func (kl *Kubelet) getHostIPAnyWay() (net.IP, error) { node, err := kl.getNodeAnyWay() if err != nil { return nil, err } return nodeutil.GetNodeHostIP(node) } </ins>

Currently we query the apiserver for the node (indirectly: ) before populating hostip on a pod. I believe this is because the node controller is responsible for calling into the cloud provider and populating the ip address on a node: This is disappointing for 2 reasons: makes using ips to identify nodes harder (). 't separate concerns sufficiently. If the reflector fails to watch nodes reliably pods won't get a hostip.
Also see .
I would argue that if we're unable to watch nodes reliably, we have many other problems. That seems like something we need to fix, rather than work around.
watch is watch, a separation of concerns between the kubelet (which is running on the node that has the hostip) and the apiserver is a different problem imo.
After discussion, we decided to just remove host IP from PodStatus. (It will still be available from "kubectl get nodes" if someone needs it.) does that sound OK with you?
I'm working on making the nodes register themselves rather than having the node controller do it for them (see which should turn into a PR later this week). Will that assuage your concerns here?
I hadn't looked at your PR. We discussed having Kubelet pull ths information from the cloud provider but thought it might not be a good idea due to (1) requiring Kubelet to understand the details of all the cloud providers, and (2) making very hard to rate limit the calls into the cloud provider. I could imagine an approach where we still use NodeController to get the node address, but the node still initiates the registration into the cluster. Not sure whether that's a good idea, though.
Kubelet won't necessarily (and really shouldn't) have the credentials necessary to contact the cloud provider. I don't remember why HostIP was to PodStatus, and I would favor removing it, except that it would be a breaking change. You could ask on google-containers@ whether anyone is relying upon it. HostIP is tricky, as hosts may have multiple addresses, in general, which is why NodeAddresses is now an array.
On GCE / AWS do you even need credentials to determine your external IP? I thought you got it from the metadata server running locally without needing to hit an Auth'd API endpoint.
HostIP wasn't the external IP AFAIK.

It's not well defined what it is, which is one reason it should be eliminated. Yes, one should be able to get the host IP from metadata, but the cloudprovider library doesn't currently do that.
I'm also in favor of removing HostIP from pods. But I don't agree with the reason in the OP (that sometimes it'll be blank when kubelet hasn't heard from apiserver)-- I think that's par for the course.
Should we schedule a meeting to discuss whether it's reasonable for Kubelets to pull their IPs directly from the cloud provider? What is the alternative -- a new API endpoint on the master where unregistered nodes can query their IP address from NodeController? Do nodes really need their IP address when they register with the master?
That sounds like a good idea (I think we can likely clear this up with a quick meeting rather than going back and forth for days in github comments).
We all agreed that we should remove HostIP from pods. Nodes do not need to provide their address(es) when they register. It's not part of the spec. They could provide whatever hostnames and/or addresses they can get through normal Linux mechanisms and/or could contact a local metadata server, but I don't think that's necessary, since the node controller will update the addresses once the node is registered, unless we want to support scenarios with no cloud provider and no external node database. In any case, we shouldn't discuss self-registration in this issue. There are other issues that cover that topic.
OK, so this issue just boils down to removing HostIP from PodStatus, and we'll discuss the rest in ?
Ping on this. e2e "Pods should get a host IP [Conformance]" is currently disabled everywhere. Bumping this to and .
Today Kubelet fills PodStatus.HostIP in generatePodStatus() in Master (e.g. NodeController) is not involved. Currently the test waits two minutes for a pod to get a HostIP Considering that we have proposed to remove this field completely, I think we should just bump up the timeout a bit and see if that fixes the test. Any other thoughts? I have also sent out email to the appropriate mailing lists to see if anyone is using PodStatus.HostIP, since we would like to get rid of it, as discussed upthread.

cc/
BTW the title of this issue says "Kubelet shouldn't require apiserver to populate host ip of pods." But it wasn't clear to me where the dependency on the API server is (unfortunately 's original comment points to a very old revision to so I'm not sure which line he was referring to). The work of finding the HostIP seems to be done by Kubelet.GetHostIP() / Kubelet.GetNode() which assumes Kubelet.nodeInfo is filled in -- is that the dependency? I couldn't find where that gets filled in. would be great if you could shed some light.
I haven't looked at this code recently but nodinfo appears to still be retrieved from a cache populated via apiserver watch:
Ah yes, that looks right, thanks (well, "right" in the sense of "that's where it gets populated," not in the sense of "it makes sense that kubelet is using code defined in the scheduler predicates for this"...) while you're here, do you have a suggestion of the best course of action? Long-term we want to get rid of PodStatus.HostIP, but short-term we need to do something about this test. How can we make populating the cache more reliable? Or should we just increase the timeout on the test? Or something else?
It looks like the kubelet already has the node address: , so assuming no one else is going to change the ip, we could just store it in the kubelet and add it for every pod? Unsure if this is a safe assumption.
I also haven't debugged this new set of test failures, so I'm not sure if it's failing for the same reason
... to make on all clouds
There isn't a new set of failures - Ike just noticed that the test is disabled, from when this issue was originally opened. could one of you comment on whether Prashanth's suggestion sounds reasonable?
I didn't parse through the entire issue since a lot of information is stale. Yes David is right, we moved to the logic of filling hostIP of Pod from apiserver to kubelet long time ago. Recording to the comment of Prashanth: Some users might have node with multiple network interfaces, and want to hostIP with specific one by updating Node object.
Fwiw I think we should turn the test on and handle things as they come up.

I think the major source of flake was handled when we did:
With respect to the long-term future of HostIP: I'm in favor of documenting the field as deprecated and asking users to contact us with their use case if they need it (in the API field documentation). As long as the field exists, I agree we should test that it's set correctly.
in this comment you wrote "Ping on this. e2e "Pods should get a host IP [Conformance]" is currently disabled everywhere. Bumping this to P1 and flake." What made you say it is disabled? I looked at Jenkins and it seems to be running it: (root)/Kubernetes%%20suite/PodsshouldgetahostIPConformance/history/ I looked in and didn't see any special handling of [Conformance]. There is an env var CONFORMANCETESTSKIP_REGEX in hack/gingko- but it doesn't seem to be set anywhere. So I don't see anything that should be causing this test to not be run. (As always there's a high likelihood I'm misunderstanding something.)
Sorry, I'm totally blind, the Jenkins history I linked to clearly shows it is being skipped. But I'm still not sure what is causing it to be skipped.
Sorry I wasn't clear; it's being skipped because it was marked pending via . Let's agree to use and labels in the future for clarity's sake.
Also see for use cases of "I need to know my local node" and possible fixes.
I'm going to remove the flake label, since has been merged and that was the suggestion here We can re-add the label if it flakes again. I'm not closing the issue, because we want to deprecate the field longer-term.


Question: <ins> //go:build !windows // +build !windows /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package editor import ( "errors" "fmt" "os" "os/exec" "path/filepath" "strings" "k8s.io/klog/v2" "k8s.io/kubectl/pkg/util/term" ) func (e Editor) args(path string) []string { args := make([]string, len(e.Args)) copy(args, e.Args) if e.Shell { last := args[len(args)-1] args[len(args)-1] = fmt.Sprintf("%s %q", last, path) } else { args = append(args, path) } return args } // Launch opens the described or returns an error. The TTY will be protected, and // SIGQUIT, SIGTERM, and SIGINT will all be trapped. func (e Editor) Launch(path string) error { if len(e.Args) == 0 { return fmt.Errorf("no editor defined, can't open %s", path) } abs, err := filepath.Abs(path) if err != nil { return err } args := e.args(abs) cmd := exec.Command(args[0], args[1:]...) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr cmd.Stdin = os.Stdin klog.V(5).Infof("Opening file with editor %v", args) if err := (term.TTY{In: os.Stdin, TryDev: true}).Safe(cmd.Run); err != nil { if errors.Is(err, exec.ErrNotFound) { return fmt.Errorf("unable to launch the editor %q", strings.Join(e.Args, " ")) } return fmt.Errorf("there was a problem with the editor %q", strings.Join(e.Args, " ")) } return nil } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--I was trying to use a different editor than notepad on windows to be used by by setting the environment variable. Doing this was non trivial and the following did not work all resulting in different errors. What happened: What you expected to happen: I should be able to use normal windows paths, either by quoting them, or not. the workaround is to use a short filename however short paths are optional and may not always be created. () How to reproduce it (as minimally and precisely as possible): try and set an editor on windows where there is a space in the name of the command. Anything else we need to know?: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: N/A this is client side issue OS (e.g. from /etc/os-release): windows 10 1803 Kernel (e.g. ): :-) Install tools: Others:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
In case anyone wants an answer, it needs two things: (or ) into system PATH environment variable EDITOR=notepad2 -w (or EDITOR=code -w)
Currently there is no way how to pass path with spaces on Windows with or environment variables. Pull Request addresses this issue. With the fix, you would be able to specify the following: and when is in your , you could do as well: or (this is current workaround for the problem)


Question: maxStartAge = time.Hour * 24 * 365 // 1 year maxStatsAge = time.Minute ) <del> // fetch node so we can know proper node memory bounds for unconstrained cgroups </del> <ins> ginkgo.By("Fetching node so we can match against an appropriate memory limit") </ins> node := getLocalNode(f) memoryCapacity := node.Status.Capacity["memory"] memoryLimit := memoryCapacity.Value()

Which jobs are failing: ci-cri-containerd-node-e2e Which test(s) are failing: Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api Since when has it been failing: 2020 10-12 09:04 PDT Since Testgrid link: Reason for failure: Capability set incorrectly. CAPNETRAW should be NETRAW. I1013 18:39:19.058] Oct 13 18:10:51.403: INFO: At 2020-10-13 18:09:34 +0000 UTC - event for stats-busybox-0: {kubelet tmp-node-e2e--cos-81--1210-0} Failed: Error: failed to create containerd task: OCI runtime create failed: starting container process caused: unknown capability "CAPCAPNET_RAW": unknown Anything else we need to know: I have no idea where this is documented. It's a best guess.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
it's a node_e2e test so /sig node
Seems like there's some dispute on who is doing the correct thing.
I would like to have a try.
please see discussion in


Question: value: 1024-4999 - name: net.ipv4.tcp_keepalive_time value: "7200" <ins> - name: net.ipv4.tcp_fin_timeout value: "60" - name: net.ipv4.tcp_keepalive_intvl value: "75" - name: net.ipv4.tcp_keepalive_probes value: "9" </ins>

Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted


Question: }) AfterEach(func() { <del> framework.RunKubectlOrDie("delete", "deployment", dName, nsFlag) </del> <ins> err := wait.Poll(framework.Poll, 2*time.Minute, func() (bool, error) { out, err := framework.RunKubectl("delete", "deployment", dName, nsFlag) if err != nil { if strings.Contains(err.Error(), "could not find default credentials") { err = nil } return false, fmt.Errorf("kubectl delete failed output: %s, err: %v", out, err) } return true, nil }) Expect(err).NotTo(HaveOccurred()) </ins> }) It("should create a deployment from an image [Conformance]", func() {

https://k8s- Failed: [] Kubectl client [] Kubectl run deployment should create a deployment from an image [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl run deployment should create a deployment from an image [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl run deployment should create a deployment from an image [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention...
https://k8s- Failed: [] Kubectl client [] Kubectl run deployment should create a deployment from an image [Conformance] {Kubernetes e2e suite}


Question: }}, expected: errors.New("kubernetes.io/foo is an invalid extended resource name"), }, <ins> { name: "invalid redeclared custom predicate", policy: api.Policy{ Predicates: []api.PredicatePolicy{ {Name: "customPredicate1", Argument: &api.PredicateArgument{ServiceAffinity: &api.ServiceAffinity{Labels: []string{"label1"}}}}, {Name: "customPredicate2", Argument: &api.PredicateArgument{ServiceAffinity: &api.ServiceAffinity{Labels: []string{"label2"}}}}, }, }, expected: errors.New("Predicate 'customPredicate2' redeclares custom predicate 'ServiceAffinity', from:'customPredicate1'"), }, { name: "invalid redeclared custom priority", policy: api.Policy{ Priorities: []api.PriorityPolicy{ {Name: "customPriority1", Weight: 1, Argument: &api.PriorityArgument{ServiceAntiAffinity: &api.ServiceAntiAffinity{Label: "label1"}}}, {Name: "customPriority2", Weight: 1, Argument: &api.PriorityArgument{ServiceAntiAffinity: &api.ServiceAntiAffinity{Label: "label2"}}}, }, }, expected: errors.New("Priority 'customPriority2' redeclares custom priority 'ServiceAntiAffinity', from:'customPriority1'"), }, </ins> } for _, test := range tests {

Currently custom predicate and priorities can be created through the . The problem is that a custom predicate/priority, like ServiceAffinity, can be multiple times in the Policy with different names, possibly with different arguments. This semantic (i.e., specifying multiple instances of the same custom predicate with different arguments) is only useful if different s were specified because in all cases of custom predicates/priorities the arguments are merge-able (for example, from multiple CheckServiceAffinity custom predicates can be merged into one set). In the world of framework plugins, we don't currently have a way to maintain those semantics. to run. The question here: do we really want to maintain the old semantics of custom predicates and priorities? in other words, do we need to change the ComponentConfig API to somehow allow specifying multiple instances of the same Plugin with different arguments/weights, or should we consider that this is not supported? I always lean towards simplifying things, so my suggestion is to not support that, but I would like to know if there are actual deployments that rely on this. /sig scheduling /cc
/cc
We discussed this in the sig meeting, and put the issue for discussion on the scheduler and kubernetes-dev mailing list, I believe this behavior is not intentional, and so we should add logic to fail the scheduler configuration when a policy includes more than one custom predicate/priority of the same type. /assign
/assign
any progress on this one?
I haven't made any progress on this yet, but it's on my priority now
In migrating custom predicates to filter plugins , I am gonna ignore the custom predicate name and use the default name.


Question: r.WatchListPageSize = 4 // Initial list should use RV=0 <del> r.ListAndWatch(stopCh) </del> <ins> if err := r.ListAndWatch(stopCh); err != nil { t.Fatal(err) } </ins> results := s.List() if len(results) != 4 { t.Errorf("Expected 4 results, got %d", len(results)) } <del> // relist should use lastSyncResourceVersions (RV=10) and since second page of RV=10 is expired, it should full list with RV=10 </del> <ins> // relist should use lastSyncResourceVersions (RV=10) and since second page of that expired, it should full list with RV=10 </ins> stopCh = make(chan struct{}) <del> r.ListAndWatch(stopCh) </del> <ins> if err := r.ListAndWatch(stopCh); err != nil { t.Fatal(err) } </ins> results = s.List() if len(results) != 8 { t.Errorf("Expected 8 results, got %d", len(results)) } <del> expectedRVs := []string{"0", "10", "10", "10"} </del> <ins> expectedRVs := []string{"0", "10", "", "10"} </ins> if !reflect.DeepEqual(listCallRVs, expectedRVs) { <del> t.Errorf("Expected series of list calls with resource versiosn of %v but got: %v", expectedRVs, listCallRVs) </del> <ins> t.Errorf("Expected series of list calls with resource versiosn of %#v but got: %#v", expectedRVs, listCallRVs) </ins> } }

/sig scalability Debugging below done by Failed test attempt: It failed with a number of "WaitingForPods: objects timed out" errors (46 failed objects in total). Sample failing pods: 580 nodes were not ready at 18:37: nodelifecyclecontroller said that gce-scale-cluster-minion-group-2-66d8's status hasn't been updated for 41 seconds. apiserver's logs show that kubelet was doing 'PUT ' every 10 seconds. Something weird happened to watch by 'shared-informers': One watch started at 18:36:52 and finished 9ms later. The next one started at 18:40:34. So it seems that there was a gap in lease watche between 18:36:52 and 18:40:34. Between 18:36:52 and 18:40:34 there is a number of leases LIST requests in the following pattern: a single /leases?limit=500 LIST request that finishes with 200 a single /leases/continue=XXX request that finishes with 400 The kube-controller-manager is retrying every 1 second up to 18:40:34.
/assign
Looking further at this it seems that number of LIST leases by kube-controller-manager increased significantly ~week ago in large scale tests (both gce and kubemark). GCE: ! Kubemark: ! This was caused by which as a side effect enabled pagination of LIST calls when starting/restarting watchers. In some cases we're getting 400 errors from api-server for this calls which results in the symptoms described above, i.e nodes becoming not-ready and pods stuck in running but not ready state for some time. GCE: ! Kubemark: !
There are actually two issues here: that we switched from non-paginated LIST calls to paginated the 10x increase in # calls is expected, but what is worrying is that we already had ~50 LISTs in every test. Such LIST happens only when watcher restarts which shouldn't happen in a healthy cluster This will be address in need to figure out why we're sometimes getting 400 response (BAD REQUEST).

/milestone v1.17
Logs when then 400 errors occurred api-server kube-controller-manager
So what happens here is: in reflector ends with an error different than so the doesn't get reset to next list is issued with set to page is fetched successfully, the next page is requested with the got from the first page and second page fails with due to error the doesn't get cleared anywhere so we go back to 2 and the 400 error keeps repeating until we get 410 (resourceVersionTooOld, on the next etcd compaction) and the is set to takes long enough for the NodeController to mark many nodes as not ready as it doesn't get any updates from watch about node leases This was indeed introduced by but the actual bug is in the which sets both and on the subsequent pages while it should set only . I will send a PR to fix it today.
FYI
Great finding.
Does this mean we have no tests that actually check if a reflector works with paging?
I see, it's the rare situation where "Watch in reflector ends with an error different than 410" that is untested. Can we set up a unit test that exercises this scenario by wrapping etcd3 storage to allow injecting an error in a watch attempt (similar to how we wrapped storage to test stale behavior in )?
I don't think it's the case - see . It clearly seems to be that pager wasn't properly tested. The problem seems to be that we're testing it with artifical stuff, whereas in real life etcd would return "bad request" errors.
Right, I'd like to see if we can trigger the error condition in a way that the restart/paging requests go to a real storage backend.
/milestone v1.18 (v1.17 code freeze)
- this is a regression introduced in 1.17, I actually believe we should fix that in 1.17.
/milestone v1.17 as noted above, this is a release blocking bug


Question: It("each node by switching off the network interface and ensure they function upon switch on", func() { // switch the network interface off for a while to simulate a network outage // We sleep 10 seconds to give some time for ssh command to cleanly finish before network is down. <del> testReboot(f.Client, "nohup sh -c 'sleep 10 && sudo ifdown eth0 && sleep 120 && sudo ifup eth0' >/dev/null 2>&1 &") </del> <ins> testReboot(f.Client, "nohup sh -c 'sleep 10 && sudo ip link set eth0 down && sleep 120 && sudo ip link set eth0 up' >/dev/null 2>&1 &") </ins> }) It("each node by dropping all inbound packets for a while and ensure they function afterwards", func() {

GCI Test Failing: (kubernetes-e2e-gci-gce-reboot-release-1.4) ([] Reboot [Disruptive] [Feature:Reboot] each node by switching off the network interface and ensure they function upon switch on) https://k8s-
Hmm. it looks like Kubelet is not detecting that network interfaces are down and updating its NodeStatus.
and commands are not in GCI. Is there any other equivalent command that we can use across GCI and other distros
Hmm... This test is testing "switching off network interface" specifically. I just try it with and . It should have the same effect.
I manually tested it with the following command on a GCI node. I observed the node get marked to NotReady and marked back to Ready. It should be okay. Just the test needs to be fixed.
Sweet! Thanks Do you want to continue owning this issue to have the test fixed?


Question: return devicePath, nil } <del> if err := os.MkdirAll(globalPDPath, 0750); err != nil { return devicePath, fmt.Errorf("fc: failed to mkdir %s, error", globalPDPath) } </del> err = b.mounter.FormatAndMount(devicePath, globalPDPath, b.fsType, nil) if err != nil { return devicePath, fmt.Errorf("fc: failed to mount fc volume %s [%s] to %s, error %v", devicePath, b.fsType, globalPDPath, err)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pod creation doesn't finish due to the following error. If the directory doesn't exist it will be created instead of returning error. What you expected to happen: Pod with FC volume works properly. How to reproduce it (as minimally and precisely as possible): Clean up under the "/var/lib/kubelet/plugins" Create a Pod with FC volume Environment: Kubernetes version (use ):Install tools: local-up- /cc
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage


Question: func DeleteCollection(r rest.CollectionDeleter, checkBody bool, scope RequestScope, admit admission.Interface) http.HandlerFunc { return func(w http.ResponseWriter, req *http.Request) { if isDryRun(req.URL) && !utilfeature.DefaultFeatureGate.Enabled(features.DryRun) { <del> scope.err(errors.NewBadRequest("dryRun is not supported yet"), w, req) </del> <ins> scope.err(errors.NewBadRequest("the dryRun alpha feature is disabled"), w, req) </ins> return }

What happened: When a dry run request is made to a modifying endpoint, and the dry-run alpha feature is disabled, it returns the error message "dryRun is not supported yet" This was true in 1.11, but in 1.12, since the feature is in alpha now, the message should say something like "the dryRun alpha feature is disabled" /kind bug /sig api-machinery


Question: prometheus.MustRegister(RuntimeOperations) prometheus.MustRegister(RuntimeOperationsLatency) prometheus.MustRegister(RuntimeOperationsErrors) <ins> prometheus.MustRegister(EvictionStatsAge) </ins> }) }

We should keep track of how stale stats used for evictions are. This will help track improvements in eviction latency, and help uncover regressions. cc If someone could assign this to me, I would appreciate that.
closing via


Question: for _, pod := range m.activePods() { pstatus, ok := m.podStatusProvider.GetPodStatus(pod.UID) if !ok { <del> klog.InfoS("ReconcileState: skipping pod; status not found", "pod", klog.KObj(pod)) </del> <ins> klog.V(4).InfoS("ReconcileState: skipping pod; status not found", "pod", klog.KObj(pod)) </ins> failure = append(failure, reconciledContainer{pod.Name, "", ""}) continue }

kubernetes 1.19.2 seeing tons of these, feels like something that should be debug or info ... not really actionable or something that it should warn about ? /sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I am also seeing this same issue what's odd is that I cannot find the container it's looking for which is probably why its giving this message, so my question would be why is it looking for this container ID?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle frozen
I agree this should be lower than warning... /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : I agree this should be lower than warning... /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think all InfoS messages under the should be moved to debug messages because it flood the log when the CPU manager enabled with the static policy.
/assign


Question: t.Fatalf("Got error GETing: %v", err) } defer resp.Body.Close() <del> if resp.StatusCode != http.StatusInternalServerError { t.Errorf("expected status code %d, got %d", http.StatusInternalServerError, resp.StatusCode) </del> <ins> if resp.StatusCode != http.StatusOK { t.Errorf("expected status code %d, got %d", http.StatusOK, resp.StatusCode) </ins> } //Test with old docker version

We've run into issues like where the Kubelet was up but not healthy. The monit monitoring for Kubelet should ensure it is "healthy" rather than just alive. We need to define what healthy means in this context and monitor it based on that. An alternate option is for the Kubelet to exit when it gets into a situation it can't serve from. This may lead to crashloops so we'd need to be careful going this route. The answer is probably a bit of both.
Courtesy of healthchecking with monit:
Health check is under this umbrella issue:
Will send a PR shortly.


Question: MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-google-containers} fi <del> if [[ "${NODE_OS_DISTRIBUTION}" == "gci" ]]; then # If the node image is not set, we use the latest GCI image. # Otherwise, we respect whatever is set by the user. NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} elif [[ "${NODE_OS_DISTRIBUTION}" == "debian" ]]; then NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${CVM_VERSION}} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} fi </del> <ins> # Sets node image based on the specified os distro. Currently this function only # supports gci and debian. function set-node-image() { if [[ "${NODE_OS_DISTRIBUTION}" == "gci" ]]; then # If the node image is not set, we use the latest GCI image. # Otherwise, we respect whatever is set by the user. NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} elif [[ "${NODE_OS_DISTRIBUTION}" == "debian" ]]; then NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${CVM_VERSION}} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} fi } set-node-image </ins> # Verfiy cluster autoscaler configuration. if [[ "${ENABLE_CLUSTER_AUTOSCALER}" == "true" ]]; then

During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm.

If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.


Question: }, { in: &api.Pod{ <ins> Spec: api.PodSpec{SchedulerName: "scheduler1"}, }, fieldSelector: fields.ParseSelectorOrDie("spec.schedulerName=scheduler1"), expectMatch:   true, }, { in: &api.Pod{ Spec: api.PodSpec{SchedulerName: "scheduler1"}, }, fieldSelector: fields.ParseSelectorOrDie("spec.schedulerName=scheduler2"), expectMatch:   false, }, { in: &api.Pod{ </ins> Status: api.PodStatus{Phase: api.PodRunning}, }, fieldSelector: fields.ParseSelectorOrDie("status.phase=Running"),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When trying to list object using field spec.SchedulerName, we are getting the below error message. Failed to list : field label not supported: spec.schedulerName What you expected to happen: To be able to fetch pods using the list and watch based on the field 'spec.schedulerName' How to reproduce it (as minimally and precisely as possible): Create a simple controller to with a list and watch method. Set the FieldSelector ( in the ListOptions structure ) in the ListFunc and the WatchFunc to 'spec.SchedulerName=custome-SchedulerName' Running this controller yields the above error. Anything else we need to know?: Description: Unable to fetch pods based on the field 'spec.schedulerName'. This was in 1.6 version, for multi scheduler support in kubernetes. We need to be able to fetch pods 1) Assigned to a particular custom scheduler. 2) Should be able to fetch the state transition of the pods scheduled by a custom scheduler using the fields selector. Whats the correct way to use 'schedulerName' fields for fetching objects? Or is there any other way for the custom scheduler to determine the above information from the kubernetes? This is an urgent issue, any quick help/suggestion will be of great help. Environment: Kubernetes version : Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-14T01:48:01Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Install tools: kubeadm
There are no sig labels on this issue.

Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
/cc
cc
SIG API Machinery doesn't own the contents of the API--we only need to be involved if the mechanism needs to be general.
Given the error message mentioned in the bug report at the top of this thread, do you have a suggestion of what we forgot to do when we the SchedulerName field to the PodSpec? i.e. to enable field selection on the schedulerName field?
/cc me


Question: return fmt.Errorf("unimplemented") } func (d *dummyStorage) Watch(_ context.Context, _ string, _ storage.ListOptions) (watch.Interface, error) { <del> return newDummyWatch(), nil </del> <ins> d.RLock() defer d.RUnlock() return newDummyWatch(), d.err </ins> } func (d *dummyStorage) Get(_ context.Context, _ string, _ storage.GetOptions, _ runtime.Object) error { <ins> d.RLock() defer d.RUnlock() </ins> return d.err } func (d *dummyStorage) GetList(_ context.Context, _ string, _ storage.ListOptions, listObj runtime.Object) error { <ins> d.RLock() defer d.RUnlock() </ins> podList := listObj.(*example.PodList) podList.ListMeta = metav1.ListMeta{ResourceVersion: "100"} return d.err

As Jordan pointed out in: watchcache treats RV=0 and RV="" the same way, even though it's not what we document: This should be fixed. /kind bug /sig api-machinery /priority important-soon
/assign
/triage accepted


Question: driver1 := &storagev1.CSIDriver{ ObjectMeta: metav1.ObjectMeta{ Name: "inline-driver-" + string(uuid.NewUUID()), <ins> Labels: map[string]string{ "test": f.UniqueName, }, </ins> }, Spec: storagev1.CSIDriverSpec{

https://k8s- CSIInlineVolumes should support ephemeral VolumeLifecycleMode in CSIDriver API 8/5 https://k8s- No response /sig storage
/triage accepted
/assign


Question: "k8s.io/kubernetes/test/e2e/framework/metrics" "k8s.io/kubernetes/test/e2e/manifest" testutils "k8s.io/kubernetes/test/utils" <ins> // ensure auth plugins are loaded _ "k8s.io/client-go/plugin/pkg/client/auth" </ins> ) var (

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: E2E keeps failing with internal Ginkgo failure of "Node 1 disappeared before completing BeforeSuite" on my PR. https://k8s- What you expected to happen: Not that. How to reproduce it (as minimally and precisely as possible): Run /retest on my PR. It's happened 5 times in a row. Anything else we need to know?: I am almost certain my change does not affect that test suite in any way.
cc
Worked on https://k8s- this morning: https://k8s-
Does this repo on an empty changelist?
Looks like it repos on ci: https://k8s-
Talked with gke dev oncall... internal dashboards look fine, doesn't seem like a gke issue.
Most interesting thing on is maybe the prow bump?
I'm seeing this too on my PR
Trying on a clean PR:
Did something bad happen between versions 342 and 346 on 1.11.0-alpha.2? CI fails to start starts: PR:
Maybe this has something to do with
I'm not sure. That was adding permissions back to nodes so they can delete themselves. It had broken downgrades, so it was reverted. It fixed this: https://k8s-
Testing the theory in
looks like no
https://k8s- so absolutely nothing changed on the test-infra side between the last pass at 9:10 and first failure at 9:55 (prow updated before 9a, same kubetest image, test-infra commit) Seems to be failing here : https://k8s- compared to passing run https://k8s- /unassign /assign
The client go auth plugins are by import. kubectl imports the gcp provider explicitly, but the e2e test apparently doesn't. is in the range when the e2e test started failing, and the description of that PR indicates it trimmed lots of dependencies. I'd look at whether the e2e test had only imported that auth plugin indirectly via one of those removed dependencies in that PR. If the e2e test requires that auth plugin, it should import it like kubectl and the kubeapiserver does:
Are you saying the e2e suite imports kubeadm code?

On Fri, May 11, 2018, 19:57 Jordan Liggitt <:
e2e imports kubeadm util: which imports preflight: which imported kubeapiserveroptions prior to : which imported: which imported the apiserver authenticator: which imported the auth plugins:
potential fix in
Thanks Great bug :^)


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" unversionedvalidation "k8s.io/apimachinery/pkg/apis/meta/v1/validation" "k8s.io/apimachinery/pkg/labels" <ins> apimachineryvalidation "k8s.io/apimachinery/pkg/util/validation" </ins> "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/kubernetes/pkg/api" apivalidation "k8s.io/kubernetes/pkg/api/validation"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: We have some cron jobs with longish names (~60 characters), and noticed that the corresponding jobs weren't being created by the cronjob controller because the name was too long. Here's what happened: There's a 63-character limit on job names (because for some reason on job creation the name gets copied into a label, where the limit is 63 chars). For example, if I try to create a job with a long name, I get the error message: By itself, this is fine. I can just make a job with a shorter name. However! If I create a cron job, the cron job controller will create jobs named . So the job's name is 11 characters longer than the job name. This means that if I create a cron job with a 63-character name, it'll result in a job name that's 74 characters, which the API will refuse to accept. This means the cron job never gets scheduled. Some possible resolutions: validation on cron job creation that refuses to let you create jobs with names 52 characters the cron job controller truncate cronjobs names remove the 63-character limit on job names (this seems difficult) There's actually a deeper problem here at play: Kubernetes' resource name limit is 253 characters. But the limit for resouce labels is 63 characters. So when a job is created, the name (which is allowed to be up to 255 chars) gets copied into a label, which has a lower limit. So the job creation fails. (these are the ) What you expected to happen: When I create a cron job, I expect it to either to create the cron job, or jobs correctly from that cron job How to reproduce it (as minimally and precisely as possible): a cronjob with a name that's more than 52 characters that it does not schedule jobs, with an error message like Anything else we need to know?: Kubernetes version: 1.7.3 Cloud provider: AWS OS: Ubuntu 16.04 Kernel: 4.4.0 example of a job definition with a long name This job definition is invalid:
There are no sig labels on this issue.

Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps
cc


Question: } func (mounter *Mounter) Mount(source string, target string, fstype string, options []string) error { <del> return nil </del> <ins> return unsupportedErr </ins> } func (mounter *Mounter) Unmount(target string) error { <del> return nil </del> <ins> return unsupportedErr </ins> } // GetMountRefs finds all other references to the device referenced // by mountPath; returns a list of paths. func GetMountRefs(mounter Interface, mountPath string) ([]string, error) { <del> return []string{}, nil </del> <ins> return []string{}, unsupportedErr </ins> } func (mounter *Mounter) List() ([]MountPoint, error) { <del> return []MountPoint{}, nil </del> <ins> return []MountPoint{}, unsupportedErr </ins> } func (mounter *Mounter) IsMountPointMatch(mp MountPoint, dir string) bool {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Calls to on unsupported platforms do not report any error and potentially misbehave. What you expected to happen: I would expect an error to be reported, that can be consumed by the caller. How to reproduce it (as minimally and precisely as possible): for and were failing on darwin, and no error was being reported. See: & Environment: OS (e.g. from /etc/os-release): I have been testing on osx/darwin, which is not a supported platform.
/sig storage
/assign


Question: PREEMPTIBLE_NODE=${PREEMPTIBLE_NODE:-false} PREEMPTIBLE_MASTER=${PREEMPTIBLE_MASTER:-false} <ins> # TODO(#26183): Provide a way to differentiate master OS distro and node OS # distro. </ins> OS_DISTRIBUTION=${KUBE_OS_DISTRIBUTION:-gci} <del> # For GCI, leaving it blank will auto-select the latest GCI image on the `dev` # channel. </del> <ins> # By default a cluster will be started with the master on GCI and nodes on # containervm. If you are updating the containervm version, update this # variable. CVM_VERSION=container-v1-3-v20160517 </ins> MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-} MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-google-containers} <del> NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-container-v1-3-v20160517} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} </del> <ins> NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-"${MASTER_IMAGE}"} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-"${MASTER_IMAGE_PROJECT}"} </ins> CONTAINER_RUNTIME=${KUBE_CONTAINER_RUNTIME:-docker} RKT_VERSION=${KUBE_RKT_VERSION:-0.5.5}

As of (specifically ), running against GCE without setting a node image defaults it to containervm, whereas before it defaulted it to the master image. For example, my config of: brings up a coreos master and then launches nodes with images. Before the above-referenced change this gave me my expected coreos nodes. Was this change intentional? The easiest fix that comes to mind for me is only doing the new containervm defaulting if is gci. cc
Yeah, this was something I was concerned too while making the change. However, the decision seems to have been that we decided to default to GCI, yet still default to the debian based containervm. Also, if we are to seriously start supporting hybrid clusters, i.e., master and nodes run on different OS distros, which seems true according to , this old assumption of defaults to needs to disappear anyways. cc/ Thoughts?
That sounds reasonable to me. We clearly didn't think about your use case whilst making this change.
I don't see the issue with defaulting to non-hybrid while supporting hybrid.. Defaulting seems totally reasonable even in a world where hybrid support is possible. A user could explicitly control that they want hybrid via setting them to different values regardless of the defaulting, or wherever hybrid is toggled on different defaulting could be done. I left a few questions on the referenced original issue which, if/when answered, might help me in making a PR to fix this for my usecase more cleanly.
Agreed. I am making a fix to this.


Question: }, }, } <del> glog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk", nodeResourceGroup, vmName) </del> <ins> glog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk(%s)", nodeResourceGroup, vmName, diskName) </ins> ctx, cancel := getContextWithCancel() defer cancel() <del> resp, err := as.VirtualMachinesClient.CreateOrUpdate(ctx, nodeResourceGroup, vmName, newVM) if as.CloudProviderBackoff && shouldRetryHTTPRequest(resp, err) { glog.V(2).Infof("azureDisk - update(%s) backing off: vm(%s)", nodeResourceGroup, vmName) retryErr := as.CreateOrUpdateVMWithRetry(nodeResourceGroup, vmName, newVM) if retryErr != nil { err = retryErr glog.V(2).Infof("azureDisk - update(%s) abort backoff: vm(%s)", nodeResourceGroup, vmName) } } if err != nil { glog.Errorf("azureDisk - azure disk detach failed, err: %v", err) </del> <ins> if _, err := as.VirtualMachinesClient.CreateOrUpdate(ctx, nodeResourceGroup, vmName, newVM); err != nil { glog.Errorf("azureDisk - detach disk(%s) failed, err: %v", diskName, err) </ins> } else { <del> glog.V(4).Info("azureDisk - azure disk detach succeeded") </del> <ins> glog.V(2).Infof("azureDisk - detach disk(%s) succeeded", diskName) </ins> // Invalidate the cache right after updating as.cloud.vmCache.Delete(vmName) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: retry azure disk attach operation produce too many API calls Not the default value is 6 which means if AttachDisk operation failed, it will retry 6 more times, when disk quota is full on a VM, it produces too many API calls. Retry attach disk behavior is not necessary since it AttachVolume operation fails, k8s attachdetach-controller will retry once a minute until it succeeds. default setting in What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.10, v1.11, v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig azure /assign


Question: gomega.Eventually(func() error { return checkMirrorPodDisappear(f.ClientSet, mirrorPodName, ns) }, time.Minute, time.Second*2).Should(gomega.BeNil()) }) }) })

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Several tests Eg. this run: Since when has it been failing: Unknown Testgrid link: Reason for failure: (happens to all the images). Pods in the e2e test suites use pull policy , , and it looks like the test during DiskPressure, resulting in kubelet removing some of the pre pulled images. Anything else we need to know: The runs after each test. This should also be the case in the test too.
/assign /sig node
These tests are now constantly failing due to missing images. This makes it hard to see if some of the stuff we do now, or have done in the last few weeks, breaks some of the tests. PR should hopefully fix this, but it is not in the 1.16 milestone, but has and . Should we get this merged into master so that we can see if if fixes the problem, or should we wait until code thaw? Any thoughts? /cc /cc /cc
/priority important-longterm
Thanks for all the work you've put into this ! I think we should add to the milestone
+1 to - thanks for your hard work :)
still has a lot of failing tests. What are the next steps? /reopen
Reopened this issue. <detailsIn response to : still has a lot of failing tests. What are the next steps? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, it has over the few last weeks become quite a mess... Looks like fixed some of the issues about image eviction problems, but there are still a few left. I think I have a solution, so ill try to submit a ~PR in a few hours~ edit: will do it tomorrow. A lot of new tests started failing 8-28, and that may be related to , but I am not totally sure..
Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! ! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above).

Ill create a new issue about all the failing tests where we can track them each together. Closing this issue, as this was only filed for the flaking becuase of the evicted images. :smile: /close cc
Closing this issue. <detailsIn response to : Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above). cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: execAffinityTestForLBServiceWithTransition(f, cs, svc) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. </del> // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { </del> <ins> ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP off [Slow] [LinuxOnly]", func() { </ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is a followup from filed separately now that we're not tracking this as part of 1.9 release. The tests have been disabled with [DisabledForLargeClusters] tag for now, and this fact will be mentioned in the scalability validation report for v1.9. This issue tracks fixing the test so the tag can be removed and the test ran again in the large cluster. /assign /kind e2e-test-failure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this e2e test still broken?
Sorry for the confusion here. Currently this e2e test is disabled on extra large cluster due to some scalability issues with the load balancer related components. At the moment the issues haven't been completely resolved yet.
ok, adding priority so that I know it's not a 1.13 blocker. /priority important-longterm
cc - this is why the relevant tests for are not part of the e2e suite.
This issue is still valid --- some action items on top of my head: Figure out a plan to make service controller properly handle operations in parallel. Check on the status for cloud providers (especially GCP, which the test ran against) on handling large number of nodes as LB backends. /remove-triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /lifecycle frozen
/reopen Reopening as is being reverted in
Reopened this issue. <detailsIn response to : /reopen Reopening as is being reverted in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
these tests are flaking in xref
cc since these were enabled by
Yeah - though these failures doesn't seem to be scale related, they seem to be test related to me. This basically means nodes without public ips: It's pretty much the same thing as (just for a different test). should be addressing these kinds of errors. [Not a release blocker for sure]
BTW - has been opened directly for this one.
... so since is closed, Can we close this and re-enable for large clusters ? (i.e. remove the Tag in that disables this test) ? I'm not up to speed on the history but 2 related notes I came across digging into related test today... 's a couple of TODOs in the code referencing this specific issue, but your above direct issue is now closed, so we can close this as well right?... And if so, we can remove the TODO on the test... -enabling the test ... not sure wether the timing is good right now (given that is open right now , which sparked my stumbling upon this issue :))
We tried to enable all of them and that caused us a bunch of issues and those had to be reverted: A couple tests were enabled later in: and those seem fine. The remaining (mostly session-affinity ones) had some issue, which so far we didn't have time to debug and I haven't heard a good explanation why those were failing. It's possible that they were improved with , but I doubt those would really cause the failures we've seen in the past. So until this is debugged (or at least understood) I'm reluctant to enabling them.


Question: NonMergingList    []MergeItem MergingIntList    []int `patchStrategy:"merge"` NonMergingIntList []int <ins> MergeItemPtr      *MergeItem `patchStrategy:"merge" patchMergeKey:"name"` </ins> SimpleMap         map[string]string }

PATCH http://IP:8080/api/v1beta3/namespaces/dev--ea6f-49cb-b5bb-/replicationcontrollers/redis-master {"spec":{"replicas":0}} answer: { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "the server responded with the status code 415 but did not return more information", "details": {}, "code": 415 } Before v0.15.0 it had worked well.
That error is returned by go-restful because it did not find any valid route with the same content type as specified in the request: Can you try setting the content-type in your request as is described here: cc
well, I've the same issue which is not fixed by setting the value, at least with the "strategic merge patch". Here's the request: Here's the response body: And here's the log trace of the request in the apiserver pod:
Come from a typo, purposed PR here:
Btw, I'm still having strange issues with sub-objects. Note that it appears only is the sub-object is already set. Here's the request: Here's the API log:
Looks like a bug in strategic merge -- it doesn't deal with pointers correctly:
cc
is it possible to have a milestone for this so I can tell if I should merger ? Thanks.


Question: // This prevents the service account update (below) triggering another token creation, if the referenced token couldn't be found in the store e.secrets.Add(createdToken) <del> liveServiceAccount.Secrets = append(liveServiceAccount.Secrets, v1.ObjectReference{Name: secret.Name}) </del> <ins> // Try to add a reference to the newly created token to the service account addedReference := false err = clientretry.RetryOnConflict(clientretry.DefaultRetry, func() error { // refresh liveServiceAccount on every retry defer func() { liveServiceAccount = nil }() // fetch the live service account if needed, and verify the UID matches and that we still need a token if liveServiceAccount == nil { liveServiceAccount, err = serviceAccounts.Get(serviceAccount.Name, metav1.GetOptions{}) if err != nil { return err } if liveServiceAccount.UID != serviceAccount.UID { // If we don't have the same service account, stop trying to add a reference to the token made for the old service account. return nil } if hasToken, err := e.hasReferencedToken(liveServiceAccount); err != nil { // Don't retry cache lookup errors return nil } else if hasToken { // A service account token already exists, and is referenced, short-circuit return nil } } // Try to add a reference to the token liveServiceAccount.Secrets = append(liveServiceAccount.Secrets, v1.ObjectReference{Name: secret.Name}) if _, err := serviceAccounts.Update(liveServiceAccount); err != nil { return err } </ins> <del> if _, err = serviceAccounts.Update(liveServiceAccount); err != nil { </del> <ins> addedReference = true return nil }) if !addedReference { </ins> // we weren't able to use the token, try to clean it up. glog.V(2).Infof("deleting secret %s/%s because reference couldn't be added (%v)", secret.Namespace, secret.Name, err) deleteOpts := &metav1.DeleteOptions{Preconditions: &metav1.Preconditions{UID: &createdToken.UID}} if deleteErr := e.client.Core().Secrets(createdToken.Namespace).Delete(createdToken.Name, deleteOpts); deleteErr != nil { glog.Error(deleteErr) // if we fail, just log it } <ins> } if err != nil { </ins> if apierrors.IsConflict(err) || apierrors.IsNotFound(err) { // if we got a Conflict error, the service account was updated by someone else, and we'll get an update notification later // if we got a NotFound error, the service account no longer exists, and we don't need to create a token for it

At least twice now, we've created a namespace and seen that the corresponding service account default secret token is never created. (The service account itself is, but not the default secret.) Namespace creation + apiserver logs logstash-ingester: namespace creation triggered service account creation but no secret logstash-broker: namespace creation triggered service account creation and secret creation Resulting service accounts Resulting secrets Controller-manager logs for broken case Controller-manager logs for working case Version Info:
Dupe of , fixed in
Actually, not a dupe. This may be a consequence of other refactors to controller logic - possibly we are missing / dropping events out of the queue. I would expect it to get fixed on the sync interval though.
We are also seeing this happening from time to time. In our case, it wasn't fixed in the next sync interval but it took actually a little bit more of 24 hours.
can we resolve this and remove it from the 1.6 milestone?
fixed in in 1.4
We just hit this exact issue running 1.5.2. Maybe the fix didn't get the root cause or there was a regression since 1.4? This should be reopened. I waited about an hour to see if the sync interval would resolve this but it did not. I fixed by deleting and recreating the namespace. I can share logs, but they are identical to what the OP noted. Version info:
Working with this seems like it's not fixed in v1.5.2 -- what should we do here? We don't have permission to re-open the ticket, but filing the same ticket with the same logs (just fresher dates) seems passive-aggressive. :-) What's your preference?
no problem, just getting back to this. can reopen this issue.
can you recreate with the controller manager running at and include controller manager log lines from ?
Fresh repo, offending namespace "foo96", with "--v=5" since we already have that logging level in place on my dev stack. This took 96 repros of a "create namespace / monitor for token creation / delete namespace" loop.

Our script's commands, starting just after 16:36:43: Controller manager log lines from The first reference to the offending namespace (foo96) and the one other entry afterwards: All controller manager log lines citing the offending namespace: For comparison, the previous successfully-created namespace: Status as of 20 minutes later:
Also, here's the apiserver logs + controller manager logs, for more action/reaction context: Namespace: foo96 (problem) Namespace: foo95 (success)
that's helpful, thanks. it looks like while the token controller is processing the creation event for the service account, it does a live lookup against etcd and fails to find the service account. what version of etcd are you running with? are you running HA etcd? do you have quorum read enabled?
We run etcd version 2.3.7 (previously saw this on 2.2.x as well). We run etcd HA, with a 3 node cluster. We do not run the apiserver with the --etcd-quorum-read flag enabled.
(Note: My specific logs above from this morning are on etcd 2.2.5, but yeah we've seen repro on 2.3.7 as well.)
yeah, the non-quorum HA etcd config is the issue. I can look into updating the token controller to tolerate that, but I'm positive there are similar issues in many controllers that assume live lookups are reliable
Ahh excellent, glad to have an explanation! So -- now that we know what to google for, we found -- where you and clayton addressed what appears to be this same problem for service accounts, and specifically noted that you wanted to avoid recommending quorum reads. Why is the answer different here? Is it because the downstream impact of non-quorum reads is too diverse to reliably retry every relevant action? (Totally fine if that's the answer, just trying to make sure we understand the reason for what appears to be an entirely opposite conclusion to the same type of problem.)

I still think controllers should be built to tolerate weak reads, and I'm willing to fix the token controller bug here, but three factors make me think this position is a losing battle: quorum read got a lot cheaper with etcd3 there are strong opinions that etcd quorum read should be on () which means that very few people are writing controllers designed to tolerate weak reads all the CI tests I'm aware of run with a single etcd or with quorum reads on
Perfect, works for us. As for fixing this particular token controller bug, I no longer have skin in that game -- your call entirely. :-)


Question: federation_api "k8s.io/kubernetes/federation/apis/federation/v1beta1" fake_federation_release_1_4 "k8s.io/kubernetes/federation/client/clientset_generated/federation_release_1_4/fake" . "k8s.io/kubernetes/federation/pkg/federation-controller/util/test" <ins> "k8s.io/kubernetes/pkg/api/unversioned" </ins> api_v1 "k8s.io/kubernetes/pkg/api/v1" <ins> extensionsv1 "k8s.io/kubernetes/pkg/apis/extensions/v1beta1" </ins> kube_release_1_4 "k8s.io/kubernetes/pkg/client/clientset_generated/release_1_4" fake_kube_release_1_4 "k8s.io/kubernetes/pkg/client/clientset_generated/release_1_4/fake" <ins> "k8s.io/kubernetes/pkg/client/testing/core" </ins> "k8s.io/kubernetes/pkg/runtime" "github.com/stretchr/testify/assert"

In PR the specific resource types are handled explicitly. Need to delete them in more generic way, in a similar way that namespace controller does.
- switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)
did fix this?
Please close or move to next-candidate with work remaining to be done.
Yes fixed by


Question: "-m", "addrtype", "--dst-type", "LOCAL", "-j", string(kubeNodePortsChain)) <ins> // If the masqueradeMark has been added then we want to forward that same // traffic, this allows NodePort traffic to be forwarded even if the default // FORWARD policy is not accept. writeLine(proxier.filterRules, "-A", string(kubeForwardChain), "-m", "comment", "--comment", `"kubernetes forwarding rules"`, "-m", "mark", "--mark", proxier.masqueradeMark, "-j", "ACCEPT", ) // The following rules can only be set if clusterCIDR has been defined. if len(proxier.clusterCIDR) != 0 { // The following two rules ensure the traffic after the initial packet // accepted by the "kubernetes forwarding rules" rule above will be // accepted, to be as specific as possible the traffic must be sourced // or destined to the clusterCIDR (to/from a pod). writeLine(proxier.filterRules, "-A", string(kubeForwardChain), "-s", proxier.clusterCIDR, "-m", "comment", "--comment", `"kubernetes forwarding conntrack pod source rule"`, "-m", "conntrack", "--ctstate", "RELATED,ESTABLISHED", "-j", "ACCEPT", ) writeLine(proxier.filterRules, "-A", string(kubeForwardChain), "-m", "comment", "--comment", `"kubernetes forwarding conntrack pod destination rule"`, "-d", proxier.clusterCIDR, "-m", "conntrack", "--ctstate", "RELATED,ESTABLISHED", "-j", "ACCEPT", ) } </ins> // Write the end-of-table markers. writeLine(proxier.filterRules, "COMMIT") writeLine(proxier.natRules, "COMMIT")

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.5.1 Environment: Cloud provider or hardware configuration: 3x Ubuntu 16.04 nodes running in VMware vSphere OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-57-generic Install tools: kubeadm Others: CNI: Canal (flannel + calico) What happened: I originally opened to make Canal CNI work when the node's FORWARD chain default policy is set to DROP. However, I realized that the DROP policy was the root cause for why my other CNI tests also were failing. When either weave and flannel CNI is used and kube-proxy has --masquerade-all enabled, pod to pod and pod to service communication fails. Specifically, kubernetes-dashboard and tiller-deploy fail to reach the kube-apiserver and enter a crash loop. Also, the kube-dns pod also crashes. <details<summarykubectl get pods --all-namespaces -o wide</summary</details<details<summarykubectl logs --namespace=kube-system kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kube-dns--gz34l</summary</details<details<summarysudo iptables -t filter -v -L FORWARD --line-numbers (197 currently has the dashboard pod scheduled. Note the 2 dropped packets.

)</summary</details</brWhat you expected to happen: Either it should be better documented that the default FORWARD chain policy should be ACCEPT or rules should be to prevent packets from hitting the default policy. As a note, Ubuntu 16.04 defaults the FORWARD chain to DROP. I assume the is why the packets are going through the FORWARD chain in the first place but without , the dashboard still can not access the kube-apiserver. One option that I suggested in is to have kube-proxy add an iptables FORWARD rule to ACCEPT packets that have been marked by KUBE-MARK-MASQ. How to reproduce it (as minimally and precisely as possible): Create a cluster with: Weave or Flannel CNI enabled on all nodes kube-proxy set to kubernetes-dashboard enabled. kubernetes-dashboard will crash often. Anything else do we need to know: is closed because I thought the issue was specific to Canal and instead opened However, this looks to be a larger issue that kubernetes itself should address.
Weave issue:
I think this is going to hit a lot of people as Docker 1.13 is now setting the FORWARD chain to drop by default if ip_forward is set. See
See also
overlap with ?
I think a solution for this issue or (which is closer to what I feel like a proper solution is) should be considered for inclusion for K8s 1.8 since it will officially support docker 1.13. The problem as I see it is that kube-proxy sets up rules to forward traffic for services (specifically I have seen the problem with NodePorts) yet does not ensure or have any requirement (to my knowledge) that the forwarded traffic be allowed when the traffic is forwarded to a host other than the one receiving the traffic. I observed this when using K8s v1.7.5 installed with kubeadm v1.7.5 and Calico v2.5. It can be remedied with the addition of but hopefully there is a better answer than doing that manually and hopefully something more specific to what is needed. I saw the comment on the Docker 1.13 validation issue linking to the configuration script used in the GCE docker images which is why the e2es are not having any issues.
If we can be more precise about which traffic we require to ALLOW, I'm OK with adding rules to open that.

I've been doing some testing with the first two options and chatted with and about these potential options for adding some forwarding rules. They all are a certainly lot more specific than the previous blanket all ALLOW. any traffic sourced or destined for the clusterCIDR. traffic that has the KUBE-MARK-MASQ set and conntrack rules to accept the follow on traffic. a third option of adding a more specific mark that is only set on specific traffic that it is known will need to be forwarded and match on that (this option would probably need the conntrack rules like the above option too) do any of those options appeal to you or do you see any problems with them?
If we add some more rules, we shouldn't put them in the top-level chain since it makes them hard to manage/clean up. Best to add them to a chain that kube-proxy owns that is jumped to from the top-level chain.
doesn't necessarily know the ClusterCIDR. I could see your proposal making sense in conjunction with .
Very good point I should have mentioned that the options I presented are assuming clusterCIDR is set. If clusterCIDR is not defined then the first option is not possible (except for the wide open option that these rules are attempting to avoid). I guess on options 2 & 3 if it was not set then a conntrack rule could be without the clusterCIDR. I would appreciate feedback on the desired behavior if clusterCIDR is not set too.
The PR that was merged solved the issue for pods inside clusterCIDR. But it doesn't handle hostNetwork pods that use the node IP, like when you want to expose the API server as a load-balanced service externally. I'm not sure why the “kubernetes forwarding conntrack pod source rule” has to be limited to the pod CIDR, since --ctstate RELATED,ESTABLISHED should already ensure it does not allow traffic that wasn't already allowed by another rule.
I'd argue that any iptables/firewalling for a pod using isn't the responsibility of Kubernetes, There's no reasonable way for k8s to actually know what interfaces a host-network container binds to, what ports it listens on, or what expectations it has. The cluster operator should probably be responsible for that networking case.
I meant explicitly regarding services where those pods are the endpoints of a service, so Kubernetes does know the address and the port.

Due to limitation of state RELATED,ESTABLISHED in FORWARD rule i.e. current implementation, the unidirectional UDP traffic apps won't work. Because packets (except first) aren't ESTABILISHED from iptables's perspective (no reply from endpoint, no handshake). Shouldn't this be something like and not
To add to comment, bidirectional UDP apps are brittle as well. Dropping or failing to reply to the first packet will prevent the app from ever receiving a second. More discussion on


Question: PROVIDER_VARS="${PROVIDER_VARS:-} ENABLE_IP_ALIASES" fi <ins> # Enable GCE Alpha features. if [[ -n "${GCE_ALPHA_FEATURES:-}" ]]; then PROVIDER_VARS="${PROVIDER_VARS:-} GCE_ALPHA_FEATURES" fi </ins> # Admission Controllers to invoke prior to persisting objects in cluster # If we included ResourceQuota, we should keep it at the end of the list to prevent incrementing quota usage prematurely. ADMISSION_CONTROL=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,Priority,ResourceQuota

The environment variable GCEALPHAFEATURES is not set correctly in the test custer.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/network: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details


Question: return devicePath, err } <del> func (util *FCUtil) DetachDisk(c fcDiskUnmounter, mntPath string) error { if err := c.mounter.Unmount(mntPath); err != nil { return fmt.Errorf("fc detach disk: failed to unmount: %snError: %v", mntPath, err) </del> <ins> func (util *FCUtil) DetachDisk(c fcDiskUnmounter, devName string) error { // Remove scsi device from the node. if !strings.HasPrefix(devName, "/dev/") { return fmt.Errorf("fc detach disk: invalid device name: %s", devName) </ins> } <ins> arr := strings.Split(devName, "/") dev := arr[len(arr)-1] removeFromScsiSubsystem(dev, c.io) </ins> return nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When user uses fc volume plugin, device file on operating system such as /dev/sdX remains after deleting a Pod. After deleting the Pod, PVC and PV, cluster admin or external-provisioner might delete the disk from the storage, therefore the block device on the worker node should be cleaned up. The photon volume plugin already has this functionality in the removeFromScsiSubsystem(). Also the gce volume plugin has similar functionality in the udevadmChangeToDrive() too. photon: echo 1 /sys/block/<dev name/device/delete gce: udevadm trigger --action=change --property-match=DEVNAME=%s Also os-brick library in the OpenStack Cinder component uses same way with photon driver. What you expected to happen: The device file on operating system such as /dev/sdX will be cleaned up after deleting PVs. Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: KVM OS (e.g. from /etc/os-release): Fedora 25 Kernel (e.g. ):Install tools: local-up- Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
I'll work on this.
Reiterating the mentions to trigger a notification: <detailsIn response to : I'll work on this. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } wg.Wait() } <del> // getPhase returns the phase of a pod given its container info. // TODO(dchen1107): push this all the way down into kubelet. func getPhase(spec *api.PodSpec, info api.PodInfo) api.PodPhase { if info == nil { return api.PodPending } running := 0 waiting := 0 stopped := 0 failed := 0 succeeded := 0 unknown := 0 for _, container := range spec.Containers { if containerStatus, ok := info[container.Name]; ok { if containerStatus.State.Running != nil { running++ } else if containerStatus.State.Termination != nil { stopped++ if containerStatus.State.Termination.ExitCode == 0 { succeeded++ } else { failed++ } } else if containerStatus.State.Waiting != nil { waiting++ } else { unknown++ } } else { unknown++ } } switch { case waiting > 0: // One or more containers has not been started return api.PodPending case running > 0 && unknown == 0: // All containers have been started, and at least // one container is running return api.PodRunning case running == 0 && stopped > 0 && unknown == 0: // All containers are terminated if spec.RestartPolicy.Always != nil { // All containers are in the process of restarting return api.PodRunning } if stopped == succeeded { // RestartPolicy is not Always, and all // containers are terminated in success return api.PodSucceeded } if spec.RestartPolicy.Never != nil { // RestartPolicy is Never, and all containers are // terminated with at least one in failure return api.PodFailed } // RestartPolicy is OnFailure, and at least one in failure // and in the process of restarting return api.PodRunning default: return api.PodPending } } </del>

Kubelet now only reports ContainerStatus. PodStatus / PodCondition is observed at master side based on a set of ContainerStatus. There is no other way to get PodIP except net container for master side. This is very confusion to the end user, , , ... Now we are finalizing on v1beta3 API, including PodStatus & ContainerStatus. We can make net container completely invisible to the end users by finishing the following steps: 1) Expose network related errors at API level 2) Generate PodStaus directly from Kubelet
3) once PodStatus generated by kubelet, PodIP can be removed from ContainerStatus.
I think reporting PodStatus rather than just container status is most important. How hard is that?
Used to be hard since there are many dependencies when I first report Status back. Now it is totally feasible. I already started to write PRs for this. :-)
One more PR required to close this one.


Question: if err != nil { return err } <ins> if m.inhibitLock != 0 { m.dbusCon.ReleaseInhibitLock(m.inhibitLock) } </ins> m.inhibitLock = lock return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: ! What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.20.1 Cloud provider or hardware configuration: OS (e.g: ): 7 Kernel (e.g. ): Linux zsm-test-2 3.10.0-1160.11.1.el7.x8664 SMP Fri Dec 18 16:34:56 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node


Question: node. If the pod is deleted and recreated on a new node, data will be lost. <del> Create the persistent volume objects in Kubernetes using </del> <ins> ##### SELinux On systems supporting selinux it is preferred to leave it enabled/enforcing. However, docker containers mount the host path with the "_svirt_sandbox_file_t_" label type, which is incompatible with the default label type for /tmp ("_tmp_t_"), resulting in a permissions error when the mysql container attempts to `chown` _/var/lib/mysql_. Therefore, on selinx systems using host path, you should pre-create the host path directory (/tmp/data/) and change it's selinux label type to "_svirt_sandbox_file_t_", as follows: ```shell ## on every node: mkdir -p /tmp/data chmod a+rwt /tmp/data  # match /tmp permissions chcon -Rt svirt_sandbox_file_t /tmp/data ``` Continuing with host path, create the persistent volume objects in Kubernetes using </ins> [local-volumes.yaml](local-volumes.yaml): ```shell

Version: Running this example on RHEL 7.2 (assume same for centos), and using hostPath, I get a permissions error when the mysql container does a . If I set selinux to permissive on the host it works. If I manually run the container with it works. I was not able to get it to work when adding to the deployment. My fix was to locate the PVs in by changing the local- file, and doing . I this comment to local- If this issue is valid I will create a quick PR to doc this selinux solution.
See pr


Question: if test.Mode == v1.PersistentVolumeBlock { // Block: check content deviceName := fmt.Sprintf("/opt/%d", i) <del> commands := generateReadBlockCmd(deviceName, len(test.ExpectedContent)) </del> <ins> commands := GenerateReadBlockCmd(deviceName, len(test.ExpectedContent)) </ins> _, err := framework.LookForStringInPodExec(pod.Namespace, pod.Name, commands, test.ExpectedContent, time.Minute) framework.ExpectNoError(err, "failed: finding the contents of the block device %s.", deviceName)

<!-- Feature requests are unlikely to make progress as an issue. Instead, please suggest enhancements by engaging with SIGs on slack and mailing lists. A proposal that works through the design along with the implications of the change can be opened as a KEP: --In block volume tests were skipped because is not using , there are more details about how these functions are used in This is a test that uses the , if tested using the it should have , in addition the test cluster should have the Things that are needed for a first time contributor a development kubernetes cluster, preferably on a public cloud instance although I'm not sure if kind can be used a CSI driver like PD CSI Driver, the storage class should be setup to use the xfs fstype the volume snapshot CRDs and controllers installed in the cluster (kubernetes-csi/external-snapshotter#usage) /sig storage /area testing /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Feature requests are unlikely to make progress as an issue. --/area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : #### What would you like to be : /area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey team! I'd like to further investigate on this issue if possible?

/assign
/area test hey sure, I've more info about how to test it, it requires these components: a development kubernetes cluster, preferably on a public cloud instance although I'm not sure if can be used a CSI driver like , the storage class should be setup to use the fstype the volume snapshot CRDs and controllers installed in the cluster ()
thanks a lot !! I might be a bit slow though, I'm pretty new to the community. I appreciate the documentation you shared, the assignation and how to replicate it, I'll test it out and if I have some doubts, I'll get back with you :D
Hey are you still working on this issue? I am interested in this
Yes, I'm still working on this, but would totally be open to doing pair-programming if you're open to it? I would benefit from you for solving the problem since I'm still pretty new to the codebase and I'm still understanding the environment.
Hey Are you still working on this, I would like to do pair-programming if possible.
Hi there definitely!! Ping me out in slack in the Kubernetes workspace with "Diego Goytia"


Question: ) const ( <del> testNodeMonitorGracePeriod = 40 * time.Second </del> <ins> testNodeMonitorGracePeriod = 50 * time.Second </ins> testNodeStartupGracePeriod = 60 * time.Second testNodeMonitorPeriod      = 5 * time.Second testRateLimiterQPS         = float32(100000)

The controller-manager node-lifecycle-controller watches node lease and update probeTimestamp , and it check node probeTimestamp is before . The node-monitor-grace-period default value is 40s. When the client connection lost, the lease watcher cannot receive new event. With the http2 health check, HTTP2PINGTIMEOUTSECONDS default is 30s, HTTP2READIDLETIMEOUTSECONDS default is 15s, it will delay 45s to kill http2 connection. During this period, node-lifecycle-controller will mark all nodes NotReady. Here are some logs When the http2 connection lost, the watch should stop as soon as possible. The reflector should relist resource, and node-lifecycle-controller should not mark all node NotReady. HTTP2PINGTIMEOUTSECONDS+HTTP2READIDLETIMEOUTSECONDS should be less than 40s. the controller-manager node-lifecycle-controller watch client srcIP:srcPort. Drop all tcp package send into srcIP:srcPort. for more than 40 seconds, all node will be marked NotReady node will be Ready after some seconds. No response <details</details<details</details<details</details<details</details<details</details<details</details
How about double-check node lease before marking node NotReady? The node-lifecycle-controller can get reference lease from apiserver with
/sig node
/sig api-machinery
/cc We have discussed this timeouts values before and myself, I also think got some metrics with lower timeouts for the HTTP2 connections. In this situation is clear that the defaults of the node watcher and the timeouts are related and having good defaults will be ideal, but the question is which values do we change ?
I think in internal network, the ping response almost return in 1 second, the HTTP2PINGTIMEOUT_SECONDS default value should be 5 seconds. And the total timeout will 35 seconds. WDYT?
/triage accepted any pointers in other places where those timeouts were changed for http2? Any notes from those discussions?
/priority important-longterm since it is not a regression I will keep it longterm priority and not higher. But we can reevaluate if we hear more reports about it.

/assign
I'm investigating problems similar to this, but I'm looking for a way to confirm this is what is happening - and if there are tips for how to recreate such a scenario, that would be very helpful. We get infrequent reports of "worker goes NotReady for a short time" incidents that appear to be triggered by events like failing over a LB sitting front of the kube-apiservers. We've done some tests with a simple client-go app and have recreated a scenario in which the app logs messages after 45 secs of "context exceeded" errors. The failures reported to us from customers don't have http2 errors in the kubelet logs, and our attempts to recreate with kubelet have never shown more more than a single error which works on subsequent retry.
Hi folks, we have a simulated recreate of the problem. This recreate involves injecting a 40 second delay in "connectivity" between the Kubernetes API server and etcd (we a 40 second delay to etcd defrag and defragged all etcd members at the same time). The connectivity between these two components is obviously critical to nodes successfully updating their leases in order to stay ready. We use 40 seconds because it matches . The test will usually cause one or more of the cluster nodes will flap their readiness state. Setting on the Kubernetes controller manager prevents the flap. Given this, I think that one or more of the Kubernetes default configurations needs to change in order to tolerate such "connectivity" problems in order to maintain healthy node state.
What's the proposal then? Do we update --node-monitor-grace-period to 50s? It seems the less risky change as the client-go defaults are more sensitive and impact the whole ecosystem, beyond kubernetes itself are you open to increase this default? It seems worth it to improve the reliability of cluster
I think we should update .
can someone send a PR to see how it will look like?
I have been following this discussion as we had encountered the bug as well. I have created a PR - Please let me know if it works
I'll add another data point from our testing. Namely, the simulated recreate noted depends on the Kubernetes controller manager (KCM) leader election configuration. Using the following leader election configuration for the KCM yields a KCM that has no leader after the delayed etcd defrag.

As a result, nodes won't be marked not ready unless the KCM leadership is restored before the nodes renew their lease. Extending the KCM leader election configuration to the following, makes the problem easier to recreate.
/assign Reassigning the issue to me


Question: ) return api.PodSpec{ <del> NodeSelector: map[string]string{ "beta.kubernetes.io/arch": runtime.GOARCH, }, </del> Containers: []api.Container{ // DNS server {

Right now we create 3 daemonsets for in , but will only give use image names for , so this feature is broken. We should only deploy one daemon set for the native architecture, but ensure it has the right label selectors, so we don't try to run pods on nodes of different architecture. We should also make sure all addons have architecture labels, and not just . If user is building heterogeneous clusters, the will need to take about deploying addons for additional architectures, we shouldn't pre-deploy multiple things knowing that they will most likely remain unschedulable.
I have the same issue as well. I tried to create a cluster with x86 (Intel NUC) and arm (Raspberry-Pi) and I see that the kube-proxy is crashing. Here's the log entry from kube-proxy Logs from kube-proxy exec user process caused "exec format error" Attached is the kube-proxy yaml file. I see that its picking up arm image Don't know how to fix this.


Question: awsRegions.Insert(region) } <del> // RecognizeWellKnownRegions calls RecognizeRegion on each WellKnownRegion func RecognizeWellKnownRegions() { for _, region := range WellKnownRegions { RecognizeRegion(region) </del> <ins> // recognizeWellKnownRegions calls RecognizeRegion on each WellKnownRegion func recognizeWellKnownRegions() { for _, region := range wellKnownRegions { recognizeRegion(region) </ins> } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: From /kind cleanup What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cloud-provider /assign