Question: }) podWorkers.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodKill, <del> RunningPod: runningPod, </del> <ins> RunningPod: orphanedPod, </ins> }) <ins> drainAllWorkers(podWorkers) </ins> if podWorkers.IsPodKnownTerminated(pod.UID) == true { t.Errorf("podWorker state should not be terminated") } if podWorkers.IsPodKnownTerminated(terminatedPod.UID) == false { t.Errorf("podWorker state should be terminated") } <del> if podWorkers.IsPodKnownTerminated(runningPod.ID) == true { t.Errorf("podWorker state should not be marked terminated for a running pod") </del> <ins> if podWorkers.IsPodKnownTerminated(orphanedPod.ID) == false { t.Errorf("podWorker state should be terminated for orphaned pod") </ins> } }

pull-kubernetes-unit TestUpdatePodWithTerminatedPod 10/8 (when it was introduced in ) No response No response Running with shows immediate failures: /sig node
/cc
/assign I will try to fix this. At present, I found that adding before the status check will definitely trigger the error
Looks like it stopped flaking? Was this fixed? /triage accepted
If the test runs fast enough, it won't failed but it's not guaranteed. fixed in
It's still flaking
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
this started flaking more on 3/21 - adding to the milestone for investigation to make sure something didn't regress here
cc
/unassign /assign


Question: ginkgo.It("should function for endpoint-Service: http", func() { config := e2enetwork.NewNetworkingTestConfig(f, false, false) ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (config.clusterIP)", config.EndpointPods[0].Name, config.ClusterIP, e2enetwork.ClusterHTTPPort)) <del> config.DialFromEndpointContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err := config.DialFromEndpointContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (nodeIP)", config.EndpointPods[0].Name, config.NodeIP, config.NodeHTTPPort)) <del> config.DialFromEndpointContainer("http", config.NodeIP, config.NodeHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err = config.DialFromEndpointContainer("http", config.NodeIP, config.NodeHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> }) ginkgo.It("should function for endpoint-Service: udp", func() { config := e2enetwork.NewNetworkingTestConfig(f, false, false) ginkgo.By(fmt.Sprintf("dialing(udp) %v (endpoint) --> %v:%v (config.clusterIP)", config.EndpointPods[0].Name, config.ClusterIP, e2enetwork.ClusterUDPPort)) <del> config.DialFromEndpointContainer("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err := config.DialFromEndpointContainer("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> ginkgo.By(fmt.Sprintf("dialing(udp) %v (endpoint) --> %v:%v (nodeIP)", config.EndpointPods[0].Name, config.NodeIP, config.NodeUDPPort)) <del> config.DialFromEndpointContainer("udp", config.NodeIP, config.NodeUDPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err = config.DialFromEndpointContainer("udp", config.NodeIP, config.NodeUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> }) // This test ensures that in a situation where multiple services exist with the same selector,

What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: // PreFilter invoked at the prefilter extension point. func (f *Fit) PreFilter(ctx context.Context, cycleState *framework.CycleState, pod *v1.Pod) (*framework.PreFilterResult, *framework.Status) { <ins> if !f.enableSidecarContainers && hasRestartableInitContainer(pod) { // Scheduler will calculate resources usage for a Pod containing // restartable init containers that will be equal or more than kubelet will // require to run the Pod. So there will be no overbooking. However, to // avoid the inconsistency in resource calculation between the scheduler // and the older (before v1.28) kubelet, make the Pod unschedulable. return nil, framework.NewStatus(framework.UnschedulableAndUnresolvable, "Pod has a restartable init container and the SidecarContainers feature is disabled") } </ins> cycleState.Write(preFilterStateKey, computePodResourceRequest(pod)) return nil, nil }

/kind feature /sig scheduling Filter will return for all nodes when is and is . This can be done is PreFilter. Executing this code in the filter will cause it to be repeated tens of times, with the worst-case scenario being repeated thousands of times. Maybe we should move this to PreFilter like:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Sounds reasonable to me :) I think you can go ahead and file a pr for this.
/assign
Sure.
Hi, If you don't mind, I can try to do this.
/assign


Question: "fmt" "sync" <ins> "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" "k8s.io/apiserver/pkg/registry/generic" "k8s.io/apiserver/pkg/storage" </ins> metainternalversion "k8s.io/apimachinery/pkg/apis/meta/internalversion" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: tried to get listing of all componentstatuses using a labelSelector query selector. I am hitting the apiServer directly. GET /api/v1/componentstatuses What you expected to happen: GET /api/v1/componentstatuses with labelSelector should limit responses to only componentstatuses with the specified label How to reproduce it (as minimally and precisely as possible): run GET /api/v1/componentstatuses with and without a labelSelector against the apiServer. Anything else we need to know?: specifically my label selector is in the form . and i can confirm everything works as expected when i hit GET /api/v1/pods Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6+IKS", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-09T13:26:51Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: IBM OS (e.g: ): Mac Kernel (e.g. ): Darwin Kernel Version 18.5.0 Install tools: Using javascript npm request to talk to api server Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
i assume its because you cant add a label to componentstatus, but they still shouldnt return if i specify a labelSelector
that's indeed a bug i presume. but what use-case makes you label-querying componentstatus?
we have a bot to periodically run a query that finds all known endpoints on the ApiServer. Then for each of the found endpoints, find all resources with our label defined. sort of an inventory tracking system.
in a similar vein, the endpoint doesnt seem to respect
/priority backlog /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /priority backlog /help Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will take a look at this unless no objections
thank you, yours!
a PR for the ComponentStatus API, could you pls kindly take a look? I will look into the metrics API probably over the weekend. Thanks
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: func (err deletedVolumeInUseError) Error() string { return string(err) } <del> func RenameDirectory(oldPath, newName string) (string, error) { newPath, err := ioutil.TempDir(filepath.Dir(oldPath), newName) if err != nil { return "", err } // os.Rename call fails on windows (https://github.com/golang/go/issues/14527) // Replacing with copyFolder to the newPath and deleting the oldPath directory if runtime.GOOS == "windows" { err = copyFolder(oldPath, newPath) if err != nil { glog.Errorf("Error copying folder from: %s to: %s with error: %v", oldPath, newPath, err) return "", err } os.RemoveAll(oldPath) return newPath, nil } err = os.Rename(oldPath, newPath) if err != nil { return "", err } return newPath, nil } func copyFolder(source string, dest string) (err error) { fi, err := os.Lstat(source) if err != nil { glog.Errorf("Error getting stats for %s. %v", source, err) return err } err = os.MkdirAll(dest, fi.Mode()) if err != nil { glog.Errorf("Unable to create %s directory %v", dest, err) } directory, _ := os.Open(source) defer directory.Close() objects, err := directory.Readdir(-1) for _, obj := range objects { if obj.Mode()&os.ModeSymlink != 0 { continue } sourceFilePointer := source + "" + obj.Name() destinationFilePointer := dest + "" + obj.Name() if obj.IsDir() { err = copyFolder(sourceFilePointer, destinationFilePointer) if err != nil { return err } } else { err = copyFile(sourceFilePointer, destinationFilePointer) if err != nil { return err } } } return } func copyFile(source string, dest string) (err error) { sourceFile, err := os.Open(source) if err != nil { return err } defer sourceFile.Close() destFile, err := os.Create(dest) if err != nil { return err } defer destFile.Close() _, err = io.Copy(destFile, sourceFile) if err == nil { sourceInfo, err := os.Stat(source) if err != nil { err = os.Chmod(dest, sourceInfo.Mode()) } } return } </del>

While debugging a node-e2e failure on Ubuntu image, I noticed that the kublet log is full of the following error messages from: The above failure results in tons of EmptyDir volumes (including both secret and confgmap) are not properly deleted, thus test failures. cc/
CC
could you share the complete kubelet logs please
you can access the kubelet log from previous run at: They have the same issue.
for the most recent test run see also
Is there a link not behind a login?
Here is one of the errors: Saad and I debugged. Our hypothesis is that: The Rename() method for EmptyDir is first creating a directory with a new name and then tries to rename the old directory to the new name. That os.Rename() is failing on Ubuntu because the directory already exists, which succeeds on other images. To verify the Ubuntu issue, we can write a small test program using os.Rename() directly on the Ubuntu image. Then the volume reconstruction logic is picking up the new directory as a new emptydir volume for the pod, which triggers another deletion round.
Some things we should improve regardless of the root cause of this: volume reconstruction logic should ignore any folders (to break the explosion of directories/volumes in cases like this). the the default method for : Understand why it does a create directory before rename. Eliminate the create directory if possible. Otherwise make sure that the created directory is deleted if subsequent rename fails. Understand why a rename is required at all before delete. Eliminate it if possible. Otherwise make sure that if delete fails, that the rename is undone.
BTW, in Go 1.8, it looks like they've changed the behavior of os.Rename: So I think we should move towards eliminating the create directory.
Does this mean the problem could be caused by a difference in the use of golang 1.7 vs. golang 1.8 for k8s binary builds?
Yes that's possible. From the development guide, go1.8 has not been verified for building k8s.
related
Is the test binary which generates the failures reported here is built with golang 1.8? I couldn't find those information through the existing logs.
From what I understand, the purpose of the rename is so that the system knows quickly that the volume has been deleted.

This is important for volumes with large content that can take a long time to delete. So I think the right set of changes that are needed are: The volume reconstruction logic should ignore directories with ".deleting~". I can make this change. The rename needs to be changed to not create a directory first. It looks like is handling this.
On Thu, Mar 23, 2017 at 01:27:30PM -0700, Dawn Chen wrote: Yes, it is. Should we build with golang-1.7 instead?
Confirmed that when building with go1.7 instead of go1.8, the node tests succeed.
Great! Thanks for the updates. We can continue carrying on our integration work. And the issue has nothing to do with Ubuntu image itself. I am leaving this open to fix the error-prone logic in EmptyDir's teardown logic, and resolving the golang 1.8 compatibility issue.
looks like a possible dup of
I've encountered this issue about a month ago and now I can confirm that this issue is caused by compiling with Go 1.8, which forbids renaming a directory into an existing directory even if the destinition is empty. So the current logic that uses call does not work properly under Go 1.8. The correct way to solve it is to change the behavior in so that it does not rely on the call to which creates a new empty directory after a successful call.
I discussed with Saad. We think that after the volume manager redesign, we don't need to do a rename at all. The volume operation executor can handle duplicate requests for volumes with the same name. I will work on testing a fix for this.


Question: <del> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kubeadm import ( "bytes" "fmt" "io" "os" "os/exec" ) // Forked from test/e2e/framework because the e2e framework is quite bloated // for our purposes here, and modified to remove undesired logging. func RunCmd(command string, args ...string) (string, string, error) { var bout, berr bytes.Buffer cmd := exec.Command(command, args...) cmd.Stdout = io.MultiWriter(os.Stdout, &bout) cmd.Stderr = io.MultiWriter(os.Stderr, &berr) err := cmd.Run() stdout, stderr := bout.String(), berr.String() if err != nil { return "", "", fmt.Errorf("error running %s %v; got error %v, stdout %q, stderr %q", command, args, err, stdout, stderr) } return stdout, stderr, nil } </del>

go test ./cmd/kubeadm/... fails with: it appears to be following a "RunCmd" pattern used by e2e tests, but unit tests should not require built binaries
can work on this
has a long discussion as to why this is. These tests only get run when gets run, which will produce the needed binary and pass it in as arg. For more information, authored this and should be able to provide more details.
Yup, the target explicitly ignores these tests: and like said, runs this script as part of its target:
then the test should have build tags to exclude it from a regular run of the package
or up for fixing that?
I thought build tags only works on the OS level or if you it would ignore it completely? Wouldnt break the command as well?
I'd also tolerate skipping the test if the built binary doesn't exist
I think that is a great solution. I think in I had initially suggested something like that. If everyone is ok with that, I will go ahead and make that change.
Sounds good, make that change and we'll see how it turns out. I'd really want these tests start working, so +1


Question: e2epod "k8s.io/kubernetes/test/e2e/framework/pod" "k8s.io/kubernetes/test/e2e/manifest" testutils "k8s.io/kubernetes/test/utils" <ins> utilnet "k8s.io/utils/net" </ins> // ensure auth plugins are loaded _ "k8s.io/client-go/plugin/pkg/client/auth"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s-(IPv6),%20master%20(dev) Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance] Since when has it been failing: Testgrid link: https://k8s-(IPv6),%20master%20(dev) Reason for failure: The tests checks for 0.0.0.0 that is IPv4 only Anything else we need to know: Probably I should check with SIG-Testing how can we add an ipv6 job and update the KEP accordingly EDIT Probably I should check with SIG-Testing how can we add an ipv6 job as part of the k/k presubmits, and update the KEP accordingly
/assign /cc
/sig scheduling

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Introduce a mechanism that allows specifying the IP protocol version used in the e2e testing. Why is this needed: This is a followup on , my understanding is that in order to discriminate between IPv4 or IPv6 we have to use the test labels and this adds complexity to the testing automation and doesn't allow to send this information to the context of the tests. Having a mechanism to specify the IP protocol will simplify the testing for IPv4 and IPv6 and will allow reusing most of the code.
/sig testing /cc /assign Does this make sense? If affirmative, what should be the best way to add this functionality? Checking the code I think that or can provide this, don't know if checking some of the cluster config parameters, ...
the testing-commons group (sig-testing) maintains the framework, but ideally the decision on how to proceed in terms of adding ipv6 detection should come from sig-network. possibly having this as agenda for a sig-network meeting is a good start, then discussing it in a testing-commons meeting. /sig network the linked issue is a big one, so i don't have all the context here. but IMO existing tests should just work for ipv6 without the requirement for a new "IPV6" tag in test names. dual-stack or protocol specific tests should have their descriptions contain the details properly (i.e. what is this test doing) we might want to add an utility function under: (there is a PR to move it BTW ) then such an utility might be useful only to tests that need to detect the protocol. one of the goals is to refactor the code in testcontext / framework, so adding this logic there is less preferred.
/area e2e-test-framework
I think that we can add an option to test_context to specify the cluster IP family the same way that exists one for the cloud provider this is simpler because when setting up the CI jobs you will know beforehand what IP family you want to test, it also will allow reusing all the e2e code for IPv6 and Dual Stack (cc: and by default will be IPv4, so it´s backward compatible.
Being able to override and specify exactly what you want the test to beleive about your cluster is nice, but ideally it should generally work without specifying more flags.

Surely we can detect this?
I was thinking during some time about this, detecting the IP family is very simple having IPv4 or IPv6 only clusters, but with the introduction of dual stack I´m afraid about the supportability and reliability of the autodetecting mechanisms in the long term. If you see the dual stack PRs there are still a lot of unknowns about the cluster behavior. My point is, let´s have something first stable that works and is simple, and once dual stack is implemented, we can replace the flag with a function in the framework . I guess that migrating from a flag variable to a function is straight forward, is just replacing by , what do you think?
in is as part of the context. do you mean adding as a function which will auto-detecting the ip family. less flags for the framework binary is preferred.
once we have this I will open a new issue to start cleaning up and , we no longer need to discriminate based on tags


Question: "github.com/ghodss/yaml" "github.com/spf13/cobra" <del> "k8s.io/apimachinery/pkg/api/errors" </del> "k8s.io/apimachinery/pkg/api/meta" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" </ins> "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/client-go/dynamic" </ins> api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/kubectl/apply/parse" "k8s.io/kubernetes/pkg/kubectl/apply/strategy"

/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.


Question: GCE_IMAGES=${GCI_IMAGE} GCE_IMAGE_PROJECT=${GCI_IMAGE_PROJECT} GCE_ZONE=us-central1-f <del> GCE_PROJECT=kubernetes-jenkins </del> <ins> GCE_PROJECT=k8s-jkns-ci-node-e2e </ins> # user-data is the GCI cloud init config file. # gci-docker-version specifies docker version in GCI image. GCE_INSTANCE_METADATA="user-data<${GCI_CLOUD_INIT},gci-docker-version=${DOCKER_VERSION}"

I want to isolate these in their own project:
Right now just the CI jobs, but after we also need to do this for the PR jobs.


Question: * [kubectl](kubectl.md)	 - kubectl controls the Kubernetes cluster manager <del> ###### Auto generated by spf13/cobra on 29-Feb-2016 </del> <ins> ###### Auto generated by spf13/cobra on 11-Mar-2016 </ins> <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/user-guide/kubectl/kubectl_run.md?pixel)]()

Tested with: Most of the simple basic starting guides direct users to create a cluster with . If I run kubectl 1.2 against a 1.1 cluster, I'll get the following behavior now I happen to know I can get the old behavior by doing but I don't expect an average user to know that or be able to find it easily. I'm mainly concerned that we will be breaking recommended flow for GKE users if they happen to use a new kubectl before their cluster's master is upgraded. We can just update our docs, but breaking a "porcelain" command seems like poor form. Other options include defaulting to the generator for a few patch releases, and/or making do some kind of version check so it can fallback to the generator if deployments/jobs are not available. thoughts?
We should update those docs and document how to achieve stable behavior: .
This would ideally use API discovery, though, to check whether the API exists.
How hard would it be to check whether Deployment existed? We had the same issue in 1.1, btw, with Job.
I'd accept a hack, also, which checked the k8s version if there wasn't enough discovery info in 1.1.
We have the same issue with Job in 1.2, also.
Fix is in


Question: ConfigError.Set(0) } } <ins> func SetNodeName(name types.NodeName) { NodeName.WithLabelValues(string(name)).Set(1) } </ins>

What would you like to be : It would be useful if the kubelet exported a metric that contained its own name as a label, for instance: . This name should be consistent with the output of . Why is this needed: This would allow us to join the kubelet metrics against metrics exported by kube-state-metrics. An example of this can be seen in We've arranged in our cluster for the labels to be set to the node's name, but this is controlled by Prometheus and depends on your scrape & relabelling config. As such, in general we can't rely on this to be the case - for instance, with the prometheus-operator the label is a IP:Port combo. /cc
/sig instrumentation
This metric as it is right now doesn’t conform to the instrumentation guidelines, so yes we should changes this as part of the metrics overhaul targeted for 1.14.
you mean to add a label in kubelet metrics ?
precisely.
/assign


Question: } } <ins> if klog.V(4).Enabled() { logNodeAndPluginScores(result, scoresMap, pod) } </ins> if len(extenders) != 0 && nodes != nil { var mu sync.Mutex var wg sync.WaitGroup

/sig scheduling A way to check the scoring of each node to figure out why a node has a high scheduling priority. proposal 1: add some debug level logs to Kube-scheduler, so we can know node score on each scoring step. proposal 2: a simple way to decrease the priority of one node(in my case a node with more than 200 pods is with the highest scheduling priority for an unknown reason) proposal 3:? All new pods are scheduled to one of five nodes in my cluster. I have checked both CPU/memory requested usage and all five nodes are bare metal and running about 150 pods or 200 pods. Pods on node request 80-93% of the memory. We have a similar issue as Currently, I think is the easiest way.
This is useful. how do you suggest we log this information? note that there are two dimensions for each pod: plugin and node score. Also, we need to limit the log length, we can't simply log the score for each node in clusters with large number of nodes.
I prefer to log only the top 5 or 10 node scores in the log. (I have not read the code yet., will dig into it later.)
We actually merged something like this that I wrote in , but had to revert it because of performance issue in large clusters. That could be a good starting point My general approach was to 1) invert the already available to map (rather than , which it currently does) and then 2) log the top 3 highest scoring plugins for each node However discovered performance issues after this in (we believe due to the many writes this process took) and it was reverted in I think we decided that some performance hit is reasonable, especially given log levels =4, but this was ~50% drop.
thanks for the information. So with current kube-scheduler, I can set log level to 10 to get every node score for each plugin. It helps in some cases.
yes, that's the current case. Though creates a lot of other noise in the logs, so I think we agreed that it is okay for us to add this kind of summary at a lower level like
I think we can use this chance to standardize the logging logic to dump scores dynamically depending on log level: for logging level ranged in , dump the scores for topM nodes/plugins.

And make it as performant as possible. for logging level ranged in , dump the scores for topN nodes, and probably show all plugins for logging level = 10, dump the scores for all nodes/plugins.
/assign
/triage accepted
/reopen Some problems that were mentioned by are not fixed yet. /cc
Reopened this issue. <detailsIn response to : /reopen After taking a glance, I can spot several issues: A consensus on the value and . This is not fully discussed yet. Further dig into the performance result. The test runs on v=1 and v=4, but actually they're the same if it's running in the context of integration framework. I'd expect to see the result comparing before/after on v=4 and v=6. Further review on the code, like adding UTs and some logic nits. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Since is merged to revert the change,
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Just FYI: this may be related to in the future.

In , we cannot add all scores to the event message (because the event msg has the limit on the length), so I am thinking we need to sort node-scores by total score and add the score of top N nodes to the event msg. And, if we do so, we can easily achieve to log the top M Node's score by using this sorted node-scores.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/lifecycle rotten
/remove-lifecycle rotten
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: verbs: - create - delete <ins> - get </ins> - list - patch - update

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig auth What happened: Testing a daemon-set spec, I deleted/recreated it multiple times. At one of the recreations, the daemon-set was not scheduled anymore, and the number of desired pods went to 0 (on a 3 nodes cluster): The kube-apiserver stderr logs shows: The kube-controller stderr logs shows: The (default) permissions for the cluster role are: Seems to be missing a permission for the resource. I edited manually the cluster-role adding the permission and then the daemonset was scheduled again. What you expected to happen: The cluster role default permissions should allow getting the resource. How to reproduce it (as minimally and precisely as possible): I tried to reproduce it again, but I'm unable to find the exact pattern to reproduce it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release):Kernel (e.g. ): Install tools: Others:
Same here (5 node cluster, k8s version 1.9.3, Ubuntu 16.04 LTS, 4.4.0-109-generic). Deamonset we were trying to create/delete is part of official elastic yaml for metricbeat installation. Problem is specific to daemonset part, other parts of manifests behave correctly on deletion/creation. (). On 1st create - all 5 nodes were populated (kubectl create -f metricbeat-). On 1st subsequent delete followed by create we were stuck with desired 0 and error in scheduler log (E0405 14:09:46. 1 ] kube-system/elk-metricbeat failed with : failed to construct revisions of DaemonSet: "elk-metricbeat-" is forbidden: User "system:serviceaccount:kube-system:daemon-set-controller" cannot get in the namespace "kube-system"). However if we changed name of deamonset (metricbeat-v1 for example) it would bring pods on 4 nodes (master missing), setting desired to 4 (instead of 5).

Again if deleted, neither that name or any previously used name would work (desired 0), while new name would spawn 4 pods (desired 4 instead 5 master missing). No stucked pods or jobs, all other not-affected deamonsets (wave-net, kube-proxy, weave-scope) all work fine.
why did this not happen in all of the recreations and only once did it fail with RBAC issue ?
maybe this is related: "However if we changed name of deamonset (metricbeat-v1 for example) it would bring pods". New name of deamonset would spaws pods, but any subsequent retry with any of previous names would fail (until we change to new name again). Once we get to we were able to spawn pods with previously given names as well, so it might be related...
DaemonSet controller only calls GET ControllerRevisions when there's ControllerRevision name collision when DaemonSet tries to create a new ControllerRevision. This is only called when the ControllerRevision cache store is stale or due to hash collisions.


Question: } ginkgo.By("deleting the pod") <del> err = podClient.Delete(ctx, pod.Name, *metav1.NewDeleteOptions(30)) </del> <ins> err = podClient.Delete(ctx, pod.Name, *metav1.NewDeleteOptions(1)) </ins> framework.ExpectNoError(err, "failed to delete pod") })

4 other Test Failures in ci-crio-cgroupv1-node-e2e-features. Test has failed 61 times (no passing result found). Earliest failure found at commit/ (Thu Oct 19 21:41 PDT). /cc { failed [FAILED] Error creating Pod: pods "pod-dualstack-host-ips9ddb9810-134c-4d35-a0c6-" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true) In [It] at: @ 10/23/23 01:38:06.024 } No response_ /sig network node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
is also failing on these tests
the test has a bug, it checks the contrary
, let's start over, 1 commit for the e2e test, other for the promotion, and let's validate everything is alright before merging


Question: package app import ( <ins> "fmt" "golang.org/x/sys/windows" "k8s.io/klog/v2" </ins> "k8s.io/kubernetes/pkg/windows/service" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Windows nodes running with high (90%) CPU usage are having calls to /stats/summary time out with "Context Deadline Exceeded". When this happens metrics-server cannot gather accurate CPU/memory usage for the node or pods running on the node which results in HPA not functioning returning <unknowninstead of values What you expected to happen: Windows nodes should be able to run with high CPU utilization without negatively impacting Kubernetes functionality How to reproduce it (as minimally and precisely as possible): a cluster with Windows Server 2019 nodes CPU intensive workloads into cluster ex: workload to put node into high CPU state node metrics with Anything else we need to know?: This issue appears to be easier to reproduce on 1.18.x nodes vs 1.16.x or 1.17.x nodes Ex: ! ! metrics-server logs show E1020 10:58:24. 1 ] unable to fully collect metrics: unable to fully scrape metrics from source kubeletsummary:akswin000000: unable to fetch metrics from Kubelet akswin000000 (10.240.0.115): Get https://10.240.0.115:/stats/summary?onlycpuandmemory=true: context deadline exceeded Kubelet logs show and Environment: Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: AKS / Azure OS (e.g: ): WIndows Server 2019 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: Docker EE version 19.3.11
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig windows /area kubelet
I was able to reproduce on 1.18.8 during a scale operation: where it fully subscribes all CPU's in Allocatable: Metric server logs: I do not see "context deadline" in kubelet but I did find some logs assocated: Once the pods are scheduled the node remains at 100% cpu and I no longer see errors but is not accurate: on the node (run after the pod deployment stabilized)
Adding to kubelete's flag and I no longer see the errors. This reserves some cpu for kubelet and other processes. We call this out in the but do not do the same for cpu.
Also, without setting system reserved CPU, it looks like does not accurately reflect CPU usage on the nodes when CPU utilization is very high. When I got some nodes into this state I did not see any error messages in logs from metrics-server and also I noticed that that querying the metrics API directly from showed recent timestamps.
I run into this again during scaling, collectiong logs for analysis here: Metric Server logs: Kubelet logs:
have you experienced this at all?
I believe has seen high CPU issues before. Not particularly on this. I've heard there were kernel issues in the past that triggered high CPU cases.
Some investigations on our end are showing that the DiagTrack service running in the infra/pause container can consume non-trivial amounts of CPU. I'll create an issue to address that problem as it might help out some here. That certainly will not eliminate the problem we are seeing tho.
is an issue for DiagTrack
The differences in behavior between 1.17. and 1.18. clusters looks like it is caused by (which was NOT backported to any prior releases) K8s version <= 1.17 behavior Prior to these changes CPU weights for containers were inferred based on limit values set in the pod specs. The observed behavior on Windows system when CPU weights were used was that under high-CPU load pods/containers were not guaranteed their scheduling time. The good news here is that system critical services like kubelet/docker/etc were still able to get enough CPU time for the nodes to function properly. K8s version = 1.18 behavior After these changes CPU limits were enforced for pods.

The observed behavior on 1.18 clusters is the when a Windows node enters a high-CPU state is that containers are guaranteed their scheduling time but host resources start to get starved. Usually this manifests as the node entering the state, metrics not working correctly (because calls to /stats/summary are timing out) or both. Suggestion / possible fixes Set the Windows process priority of the kubelet to  to help ensure it gets CPU time Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) Set reserve CPU with with --system-reserves=[cpu=500m] This needs to scaled based on number of CPUs on host based on my experimentations Currently guidance on for Windows only mentions reserving system-memory It is still possible to over-commit CPU and starve kubelet/docker/HCS if users schedule pods with mis-matching requests and limits Update documentation.
Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) I can work on this, I am wondering if we should do the same for CRI(like docker) and CNI processes.
Sure, if you want to work on this go ahead. I think we should probably try to make this be toggleable via a kubelet flag. I believe there are other windows specific flags already implemented in the kubelet (like one to run as a windows service IIRC)
docker/containerd should probably have a flag that allows them to run as high priory as well but would likely be different issues in the respective projects
/reopen I don't think this was ever fully addressed
Reopened this issue. <detailsIn response to : /reopen I don't think this was ever fully addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
also related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: * **Kubernetes Web Interface** ([ui.md](ui.md)): Accessing the Kubernetes web user interface. <del> * **Kubecfg Command Line Interface** ([cli.md](cli.md)): The `kubecfg` command line reference. </del> <ins> * **Kubectl Command Line Interface** ([kubectl.md](kubectl.md)): The `kubectl` command line reference. </ins> * **Roadmap** ([roadmap.md](roadmap.md)): The set of supported use cases, features, docs, and patterns that are required before Kubernetes 1.0.

It's causing problems and is kind-of GCE dependent anyway, we should just use the kubecfg binary, and document the environment variables it uses.
I think its beneficial to keep something, but remove the GCE dependent pieces, i.e. "external-ip" piece.
I put together some instructions here:
We can remove this when kubecfg is deprecated .
Reopened since it still needs to be removed. /cc
Doh. Didn't mean to declare this fixed in that PR.
Closing this (again) as it is a dupe of


Question: "k8s.io/apimachinery/pkg/util/uuid" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" <ins> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" </ins> imageutils "k8s.io/kubernetes/test/utils/image" admissionapi "k8s.io/pod-security-admission/api" )

Example failure - Serial node e2e tests This is new test introduced in v1.27 No response No response No response /sig node
It seems the issue is that a previous test that ran had a pod around which was not terminated, namely The test started around but (which was created by pod-conditions-test), was fully terminated only at which is after this test started: Due to the pod not being fully terminated when the test is being run, the metric assertions fail.
/triage accepted


Question: return nil } <del> func (ds *dockerService) determinePodIPBySandboxID(uid string, sandbox *dockertypes.ContainerJSON) string { </del> <ins> func (ds *dockerService) determinePodIPBySandboxID(uid string) string { </ins> glog.Warningf("determinePodIPBySandboxID is unsupported in this build") return "" }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: PR tries to reduce CNI calls for sandboxed containers: case is for Windows Server 2016 and for Windows Server version greater than 1709 which use Hyper-V containers. Currently, the CNI on Windows fetches the IP from the containers within the pods regardless of the mode. This should be done only in the non-sandbox mode where the IP of the actual container will be different than the IP of the sandbox container. from the same pod will share the network details of the sandbox container. when this mode is supported. However, for Windows 1709 and 1803 we didnt have the Namespace support in Windows which we are bringing in RS5. So we need this second call to replicate the DNS registry key to the Workload container. This is a platform requirement and removing this will break DNS for the workload containers. IP/Gateway/MAC is set separately than DNS. (Refer ) In Rs5 once we move the namespace model, we will update CRI and CNI to add endpoints and add it to the namespace. So the PR breaks dns for windows containers. We should revert it. /sig windows /kind bug
Since is headed for 1.11, this issue should be similarly milestone 1.11 labeled and approved. /milestone v1.11 /status approved-for-milestone
You must be a member of the github team to set the milestone. <detailsIn response to : Since is headed for 1.11, this issue should be similarly milestone 1.11 labeled and approved. /status approved-for-milestone Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
You must be a member of the github team to add status labels.
/milestone v1.11 /status approved-for-milestone
if you can drop an update here on status of the PR revert of PR , and if there's anything additional that might be happening like a re-fix of what was being attempted in the earlier PR, the 1.11 release team would appreciate the additional context.

PR has been reviewed by but still waiting for sig-node's approval
/priority critical-urgent /status in-progress
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: {% set override_deb='' %} {% set override_deb_sha1='' %} {% set override_docker_ver='' %} <ins> # Ubuntu presents as os_family=Debian, osfullname=Ubuntu {% elif grains.get('cloud', '') == 'aws' and grains.get('os_family', '') == 'Debian' and grains.get('oscodename', '') == 'vivid' -%} # TODO: Get from google storage? {% set docker_pkg_name='docker-engine-1.8.3' %} {% set override_docker_ver='1.8.3' %} {% set override_deb='docker-engine_1.8.3-0~vivid_amd64.deb' %} {% set override_deb_url='http://apt.dockerproject.org/repo/pool/main/d/docker-engine/docker-engine_1.8.3-0~vivid_amd64.deb' %} {% set override_deb_sha1='f0259b1f04635977325c0cfa7c0006e1e5de1341' %} </ins> {% else %} {% set docker_pkg_name='lxc-docker-1.7.1' %} {% set override_docker_ver='1.7.1' %}

GCE updated docker, but only via a base-image update. AWS should match versions of docker, to slipstream off all the testing that GCE gets. This will be slightly complicated because docker 1.8.3 isn't in the bucket AFAICT. Thanks to for the tip.
This is done by


Question: ) const ( <del> // emptyUniquePodName is a UniquePodName for empty string. emptyUniquePodName types.UniquePodName = types.UniquePodName("") </del> <ins> // EmptyUniquePodName is a UniquePodName for empty string. EmptyUniquePodName types.UniquePodName = types.UniquePodName("") </ins> <del> // emptyUniqueVolumeName is a UniqueVolumeName for empty string emptyUniqueVolumeName v1.UniqueVolumeName = v1.UniqueVolumeName("") </del> <ins> // EmptyUniqueVolumeName is a UniqueVolumeName for empty string EmptyUniqueVolumeName v1.UniqueVolumeName = v1.UniqueVolumeName("") </ins> ) // NestedPendingOperations defines the supported set of operations.

During testing a AWS cluster, found AWS volume once was pending for detach about 30 minutes. According to the log, the following happened, controller issues a detach operation after detach response returns, it periodically checks the status until it reaches "detached". However, in this test, the status is "busy" all the time. after reaching the timeout 30 mins, detach function returns error because the status is not "detached" The following is the log related to the failed test I0915 18:29:46. 5 ] Verified volume is safe to detach for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-". E0915 18:59:54. 5 ] Error detaching volumeID "vol-": Timeout waiting for volume state: actual=busy, desired=detached E0915 18:59:54. 5 ] Operation for """" failed. No retries permitted until 2016-09-15 18:59:55. +0000 UTC (durationBeforeRetry 500ms). Error: DetachVolume.Detach failed for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-" with: Timeout waiting for volume state: actual=busy, desired=detached
Thoughts on what the ideal timeout should be. Too short would be bad because it would give a false-negative.
I know this is from a test but I'm seeing the same problem on a production cluster (1.4.8). I'm not sure exactly what triggers it yet. Kubelet only unmounts the pod directory, not the mount in "aws-ebs/mounts". As a result, the EBS gets stuck in "busy" state when detaching since it is still mounted to the VM. Manually unmounting the EBS allows it to get unstuck and detach safely. In controller manager I see: It then proceeds to get stuck detaching (volume goes to busy state), after which it starts trying to force detach which of course hangs indefinitely.

Inspecting the node directly, I see it's still mounted which is why detach hangs: So something in kubelet is messed up and not unmounting both mount points correctly.
I think this might be the same issue as
On kubelet, it looks like the volume failed to mark as mounted, which is why it subsequently got stuck. This may have to do with the fact this pod () got rolled back while waiting for the previous deployment to succeed.
^ Looks like it might be a race condition between the controller manager and reconciler: manager attaches volume starts attachment process (pod rollback/delete) controller manager thinks its safe to detach, so removes volume from node's volume list, then tries to detach mounts the volume as part of the ongoing attach/mount process from (2) tries to update the mount list, this fails because something (reconciler?) has removed the volume from attached list gets stuck in (busy) state detaching, as its mounted to the kubelet node. it never gets unmounted again in kubelet because state doesn't reflect it is mounted.
Looks like there exists race conditions between external updates of and 's asynchronous kick off of operations like MountVolume, which is the case I'm seeing. Seems like a lot of these edge cases could crop up here - maybe reconciler should lock changes to the desiredStateOfWorld while it's running?
thank you for providing detailed information and analysis. I think there is a race condition between asynchronous operation MountVolume and updating actualStateOfWorld directly by reconciler I will work on a fix. Please let me know if you have any issue/comments.


Question: }, mounts.GetVolumes(kubeadmconstants.KubeScheduler)), } <ins> // Some cloud providers need extra privileges for example to load node information from a config drive // TODO: when we fully to external cloud providers and the api server and controller manager do not need // to call out to cloud provider code, we can remove the support for the PrivilegedPods if cfg.PrivilegedPods { staticPodSpecs[kubeadmconstants.KubeAPIServer].Spec.Containers[0].SecurityContext = &v1.SecurityContext{ Privileged: utilpointer.BoolPtr(true), } staticPodSpecs[kubeadmconstants.KubeControllerManager].Spec.Containers[0].SecurityContext = &v1.SecurityContext{ Privileged: utilpointer.BoolPtr(true), } } </ins> return staticPodSpecs }

Dear All: I'm not sure if I'm right. When i use whith .I found controller-manager logs Someone can help me? Thanks a lot.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Please give more information about your set up cc
Sorry for report later. My openstack didn't support metadata service,but support config-drive service. More info:
Thanks Based on this, I think it's an openstack cloudprovider problem more than a kubeadm problem...
Thank you for help me. I will change the label.
/area cloudprovider
/remove-kind support
Should be fixed with you can specify a search order for metadata and config drive /close
Specifying a search order doesn't really solve the problem, especially if the metadata api isn't available which is what ZhenyangZhao said . I'm getting the same issue and since the config drive is my only option at the moment.
Like for this to be /open
can you please provide logs/versions for your problem in your environment? Thanks, Dims
Let me know if you need anything else Kubernetes version: 1.8.4 Docker version: 17.09.0-ce Linux: CentOS 7.4 Looking at the docker container log for the api-server (I don't have problems with the controller-manager):
and share the following:
Hi! Thank you for filing this issue. We appreciate your interest in Kubernetes and kubeadm. In order to make the kubernetes/kubernetes issue tracker more usable, we're routing issues to different repositories with different focuses. I think this issue belongs in the kubeadm repository. Please move this issue to the in order for the kubeadm team to be able to track this efficiently. If you disagree with my assessment of moving this issue, please speak up and/or reopen the issue. Finally, please try to reproduce the bug with the latest Kubernetes version, it might as well already have been fixed :) Thank you and have a good day!


Question: }} } <ins> func getConditionsByType(list []batch.JobCondition, cType batch.JobConditionType) []*batch.JobCondition { var result []*batch.JobCondition for i := range list { if list[i].Type == cType { result = append(result, &list[i]) } } return result } </ins> func (pb podBuilder) name(n string) podBuilder { pb.Name = n return pb

If the user update , duplicate condition: ! Only one Failed condition Create Job and wait it marked as Failed Update the Failed condition to False No response <details</details<details</details<details</details<details</details<details</details<details</details
It's not a bug. The is a list, so it can contain many conditions with different probe timestamps. Also, we should not update the fields.
But should we avoid the duplicate Type's conditionIn my opinionthis will confuse the user. According to kubelet, it will rewrite the 'Ready' condition in pod, and take is as managed only by kubelet. Should we did this for job Failed condition? Refer:
/sig apps
It is a bug. We should only have one condition for each type. /triage accepted
/assign
/assign
/assign
I have reproduced the issue locally and intend work on the fix
/priority backlog Since the user is manually modifying the status, this is not an inherent bug in the controller. Still worth avoiding duplicates.


Question: cStatus.Message = tMessage } } statuses[i] = cStatus }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Pod status started null, started, 01 Jan 0001 Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:48:59Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:38:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}` Environment: Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Install tools: kubeadm 1.6 What happened: When a pod is created and starts running, shows the correct container start time in Containers -containername -State -Started. After the container is terminated, the start time changes to . In addition, shows that is . What you expected to happen: I'm expecting the start time to not change after the pod is terminated, so that afterwards I could check how long a time the container execution took. How to reproduce it (as minimally and precisely as possible): Have a running kubernetes cluster, and create a pod with the following command: kubectl create -f - <<EOF apiVersion: v1 kind: Pod metadata: name: sleep spec: containers: name: sleep image: ubuntu command: ["sleep"] args: ["30"] restartPolicy: Never EOF This created a container that sleeps for 30 seconds and quits.

During those 30 seconds, check the pod status with : ... State: Running Started: Wed, 17 May 2017 10:25:22 +0300 ... After 30 seconds, check the status of the terminated container with again: ... State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Wed, 17 May 2017 10:25:52 +0300 ... With we see that the container's start time is null: ... "state": { "terminated": { "containerID": "docker://ca0dbe0dab3bb91ad0b05ac167e44d5fbf805f736ef98f61b9e106ba4456667f", "exitCode": 0, "finishedAt": "2017-05-17T07:25:52Z", "reason": "Completed", "startedAt": null } } ... Anything else we need to know: I need to measure container run durations and this bug prevents me from doing so.
/cc PR to fix this:


Kubernetes version Serverversion 1.5.2 (running Hyperkube v1.5.2coreos.1 ()) Environment: Bare-Metal Coreos 1235.6.0 (Kernel 4.7.3-coreos-r2) What happened: kube-controller-manager with ecdsa key fails to start F0127 09:59:04. 1 ] error running controllers: Error reading key for service account token controller: error reading private key file data does not contain a valid RSA or ECDSA private key The key is valid (and works with e.g. the api-server) Command used to create the key:
the issue is the block. removing that (invoke openssl with to omit it from the start) will allow the key to be consumed. will fix in 1.6 to ignore that block when reading the private key block


Question: The controller manager for Replication Controllers and other future controllers act on behalf of a user via delegation to perform automated maintenance on Kubernetes resources. Their ability to access or modify resource state should be strictly limited to their intended duties and they should be prevented from accessing information not pertinent to their role.  For example, a replication controller needs only to create a copy of a known pod configuration, to determine the running state of an existing pod, or to delete an existing pod that it created - it does not need to know the contents or current state of a pod, nor have access to any data in the pods attached volumes. <del> The Kubernetes pod scheduler is responsible for reading data from the pod to fit it onto a minion in the cluster.  At a minimum, it needs access to view the ID of a pod (to craft the binding), its current state, any resource information necessary to identify placement, and other data relevant to concerns like anti-affinity, zone or region preference, or custom logic.  It does not need the ability to modify pods or see other resources, only to create bindings.  It should not need the ability to delete bindings unless the scheduler takes control of relocating components on failed hosts (which could be implemented by a separate component that can delete bindings but not create them).  The scheduler may need read access to user or project-container information to determine preferential location (underspecified at this time). </del>  No newline at end of file <ins> The Kubernetes pod scheduler is responsible for reading data from the pod to fit it onto a minion in the cluster.  At a minimum, it needs access to view the ID of a pod (to craft the binding), its current state, any resource information necessary to identify placement, and other data relevant to concerns like anti-affinity, zone or region preference, or custom logic.  It does not need the ability to modify pods or see other resources, only to create bindings.  It should not need the ability to delete bindings unless the scheduler takes control of relocating components on failed hosts (which could be implemented by a separate component that can delete bindings but not create them).  The scheduler may need read access to user or project-container information to determine preferential location (underspecified at this time). </ins>

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
/assign /area kubeadm
/sig cluster-lifecycle
i went trough all the flakes we have at the moment and adjusted a number timeouts. here is the PR:
was green for a bit since merge, but noticed a timeout on this morning. Do you think it is worth bumping the timeout there as well? It looks like the timeout was quite a large outlier compared to the successful runs (more than double the duration), so maybe we should address underlying problem.
lets wait for some more singlal before bumping the timeouts on init. locally for me takes a couple of minutes, so i would like to know if this is a temporary GCE/networking issue.
actually something is not right, because in this job: also this job showed that even 10m is not enough for it errored with: but looking at the log timestamps only ~2 minutes have passed. i will have another look later today or tomorrow.
/retitle [Flaky Test] kubeadm-kinder-upgrade-1-18-master (ci-kubernetes-e2e-kubeadm-kinder-upgrade-1-18-master) /priority important-soon /remove-kind failing-test /kind flake the test seems to be timeouting occasionally now. this appeared to be a temporary network speed issue. in the meantime i have created this PR which attempts to catch network flakes early and reduce overall network traffic:
the kubeadm e2e jobs started requiring more than 40 minutes: https://k8s- so a lot of test jobs are now failing. this could be due to: resource limits in Prow more conformance tests were during the weekend or next week i will try bumping all jobs to ~60 minutes.
/remove-priority important-soon (prio is already critical)
xref
hi, the jobs seem to be now at the state where they were before the ticket was logged - i.e. mostly green. the Prow issue that was discussed in is definitely a factor, so i'm not going to bump the jobs to 60minutes for now.

we do still see 2 occasional types of flakes: one of them is known and we don't have a fix for it as it's complicated. for the other one i'm going to send a PR and ping you for a quick lgtm. thanks.
thanks for following up! I will make sure to take a look at your incoming PR :+1:
i haven't seen the same failure in the last 10 runs: https://k8s- please re-open this ticket if needed. thanks.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am seeing some pretty regular failures here again. Could you take a look? /reopen
Reopened this issue. <detailsIn response to : I am seeing some pretty regular failures here again. Could you take a look? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the job is taking more than 40 minutes. so this is related to:
The runs in last few days have been well under 60m, except for
i'm suspecting slow CPU/network for some of the recent failures.
/remove-sig testing SIG testing does not own kinder or upgrades
looks like the most prominent recent flakes are caused by the inability to prepare the node image or upgrade all nodes in less than 10 minutes. i will send a PR to increase these timeouts.
Is that test now named ?
yes. i see some occasional flakes of not being able to complete the conformance suite in 25 minutes (parallel).
/retitle [Flaky Test] kubeadm-kinder-upgrade-1-18-1-19 (ci-kubernetes-e2e-kubeadm-kinder-upgrade-1-18-1-19)
i can conform that the suite requires mostly more than 25 minutes in later releases (either due to more tests or slower infra). this PR increases the e2e test suite in kubeadm to 35 minutes and will close this ticket: we should create new / separate tickets with flaky tasks in the job.


Question: By("Trying to launch a pod that with NodeAffinity setting as embedded JSON string in the annotation value.") labelPodName := "with-labels" <del> testPodPath := string(framework.ReadOrDie("test/e2e/testing-manifests/node-selection/pod-with-node-affinity.yaml")) </del> <ins> testPodPath := "test/e2e/testing-manifests/node-selection/pod-with-node-affinity.yaml" </ins> framework.RunKubectlOrDie("create", "-f", testPodPath, fmt.Sprintf("--namespace=%v", ns)) // check that pod got scheduled. We intentionally DO NOT check that the

https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity

setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s-

Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the

annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
can you please take a look at this?
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
- this test is completely broken in serial suite and is P0 for last 3 days.
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
- do you think is the correct owner for this test?

https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
- I'm going to make you an owner of this test (), as you're an author if github is to be believed. FYI: assigning random people to tests doesn't work (as can be seen in this example).
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
thanks for reminding, I'm taking look at this.

https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
Found the root cause
- awesome job! Thanks!
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}


Question: activeStatus = "ACTIVE" errorStatus  = "ERROR" <ins> ServiceAnnotationLoadBalancerFloatingNetworkId = "loadbalancer.openstack.org/floating-network-id" </ins> ) // LoadBalancer implementation for LBaaS v1

Is this a BUG REPORT or FEATURE REQUEST?: Feature request /kind feature What happened: Currently we have kubernetes cluster installed, in node machines we have one subnet and two routers attached. One router for external network and one router for internal network. However, when we create type: loadbalancer services in kubernetes, we have to hardcode floatingnetwork id to only one of those (external or internal). What you expected to happen: I except that I could define from which floatingnetwork loadbalancer is taking its floating ip address in each service. One possible solution could be that if floatingnetworkid is defined like currently - that networkid is default one. Then we could have possibility to override that in each service configuration. Like if we do not want use default floating-pool we could change it: I do not know does this even work in openstack, it might need some manual route settings in lbaas. Environment: Kubernetes version (use ): 1.6 Cloud provider or hardware configuration**: openstack OS (e.g. from /etc/os-release): centos cc
Exposing first to external network so the external floatingip is working. Then update floating-networkid in kubernetes cloud-config in all nodes+masters and restart all machines. Exposing to internal network However, this last curl did not work from outside openstack. There is some route issue with it, if I try that outside openstack the result is: tcpdump from node the external one which works 192.168.1.12 is lbaas fixed ip: tcpdump from node the internal one which does not work 192.168.1.13 is lbaas fixed ip: This might mean that lbaas is using default router (which is external) to answer back to client, so the route in lbaas should be modified. No idea is that possible.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /sig network
I have defined host route to subnet, which is 10.222.0.0/16,192.168.1.4 this row basically means that all connections toward 10.222.0.0/16 should use 192.168.1.4 router interface, which is internal router. However, openstack lbaas is ignoring these host routes so before that is fixed there is no sense to code this feature to kubernetes. edit. looks like it is fixed that fix is not to our openstack yet. After that fix this is this my idea should work. Now the question is how we should implement this multiple floatingnetworkids to kubernetes. I can make PR but first I would like to discuss how it should be done.
Sounds good. cc
the problem what I am thinking currently is that I would like to have possibility to specify floating-pool in each service. Some service people may want expose service to external network and some service to internal network. However, the service api spec is quite hardcoded or can I add variable there? I am planning to add variable like there.


Question: GCE_IMAGE_PROJECT=kubernetes-jenkins INSTALL_GODEP=true CLEANUP=true <del> GINKGO_FLAGS= </del> <ins> GINKGO_FLAGS=--ginkgo.skip=FLAKY </ins> SETUP_NODE=false

As of now node e2e requires all the tests to pass. There are tests like the container conformance tests which have been flaky in the past - , , We need ability to mark certain tests as flaky and not run them as part of merge queue.
Ginkgo flags makes it possible to skip tests. We can use a prefix to not run flaky tests in our existing jenkins CIs.
Closing via


Question: deps = [ "//pkg/apis/core:go_default_library", "//pkg/apis/core/v1:go_default_library", <ins> "//pkg/controller/volume/persistentvolume/util:go_default_library", </ins> "//pkg/kubeapiserver/admission:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", </ins> "//staging/src/k8s.io/apiserver/pkg/admission:go_default_library", "//staging/src/k8s.io/cloud-provider:go_default_library", "//staging/src/k8s.io/cloud-provider/volume:go_default_library",

API server latency for PV creation depends on underlying cloud API latency. We have admission plugin that examines incoming PV and adds topology labels and node selector: When the cluster (and cloud) are under pressure, e.g. lot of PVCs are created and dynamically provisioned at one time and attached to nodes, API server latency increases. We've seen cases when we ran out of cloud API request quota and the admission plugin needed to sleep for couple of seconds (on AWS). There should be an easy fix: check, if the created PV already has all necessary labels and topology information and ask the cloud for them only when they're missing. Dynamically provisioned volumes already have these labels. The admission plugin will then ask the cloud only for manually provisioned PVs, which we believe is just a fraction of all PVs in the cluster. There is one glitch: previously, when user created a PV with wrong topology labels, the admission plugin corrected them. With the fix suggested above, this PV won't be corrected. /sig storage cc
Implementation of the above:


Question: match: configs[DistrolessIptables] = Config{list.BuildImageRegistry, "distroless-iptables", "v([0-9]+).([0-9]+).([0-9]+)"} - name: "registry.k8s.io/go-runner: dependents" <del> version: v2.3.1-go1.22-bookworm.0 </del> <ins> version: v2.3.1-go1.22.0-bookworm.0 </ins> refPaths: - path: build/common.sh match: __default_go_runner_version=

https://prow.ppc64le- Tests are not getting executed due to improper go version. It has been failing since past 4 days. No response No response No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This seems to be failing due to improper version addition in dependency files where is instead of at some places. I will raise a PR for it.
/assign
fixing that


Question: package volume import ( <del> "io" "io/ioutil" "os" filepath "path/filepath" "runtime" </del> "time" <del> "github.com/golang/glog" </del> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types"

While debugging a node-e2e failure on Ubuntu image, I noticed that the kublet log is full of the following error messages from: The above failure results in tons of EmptyDir volumes (including both secret and confgmap) are not properly deleted, thus test failures. cc/
CC
could you share the complete kubelet logs please
you can access the kubelet log from previous run at: They have the same issue.
for the most recent test run see also
Is there a link not behind a login?
Here is one of the errors: Saad and I debugged. Our hypothesis is that: The Rename() method for EmptyDir is first creating a directory with a new name and then tries to rename the old directory to the new name. That os.Rename() is failing on Ubuntu because the directory already exists, which succeeds on other images. To verify the Ubuntu issue, we can write a small test program using os.Rename() directly on the Ubuntu image. Then the volume reconstruction logic is picking up the new directory as a new emptydir volume for the pod, which triggers another deletion round.
Some things we should improve regardless of the root cause of this: volume reconstruction logic should ignore any folders (to break the explosion of directories/volumes in cases like this). the the default method for : Understand why it does a create directory before rename. Eliminate the create directory if possible. Otherwise make sure that the created directory is deleted if subsequent rename fails. Understand why a rename is required at all before delete. Eliminate it if possible. Otherwise make sure that if delete fails, that the rename is undone.
BTW, in Go 1.8, it looks like they've changed the behavior of os.Rename: So I think we should move towards eliminating the create directory.
Does this mean the problem could be caused by a difference in the use of golang 1.7 vs. golang 1.8 for k8s binary builds?
Yes that's possible. From the development guide, go1.8 has not been verified for building k8s.
related
Is the test binary which generates the failures reported here is built with golang 1.8? I couldn't find those information through the existing logs.
From what I understand, the purpose of the rename is so that the system knows quickly that the volume has been deleted.

This is important for volumes with large content that can take a long time to delete. So I think the right set of changes that are needed are: The volume reconstruction logic should ignore directories with ".deleting~". I can make this change. The rename needs to be changed to not create a directory first. It looks like is handling this.
On Thu, Mar 23, 2017 at 01:27:30PM -0700, Dawn Chen wrote: Yes, it is. Should we build with golang-1.7 instead?
Confirmed that when building with go1.7 instead of go1.8, the node tests succeed.
Great! Thanks for the updates. We can continue carrying on our integration work. And the issue has nothing to do with Ubuntu image itself. I am leaving this open to fix the error-prone logic in EmptyDir's teardown logic, and resolving the golang 1.8 compatibility issue.
looks like a possible dup of
I've encountered this issue about a month ago and now I can confirm that this issue is caused by compiling with Go 1.8, which forbids renaming a directory into an existing directory even if the destinition is empty. So the current logic that uses call does not work properly under Go 1.8. The correct way to solve it is to change the behavior in so that it does not rely on the call to which creates a new empty directory after a successful call.
I discussed with Saad. We think that after the volume manager redesign, we don't need to do a rename at all. The volume operation executor can handle duplicate requests for volumes with the same name. I will work on testing a fix for this.