Question: if err != nil { return nil, err } <ins> if !tpExists { glog.Infof("Target pool %v for Service %v/%v doesn't exist", loadBalancerName, apiService.Namespace, apiService.Name) } </ins> // Now we get to some slightly more interesting logic. // First, neither target pools nor forwarding rules can be updated in place -

Restarting master causes a recreation of fwrule and target pool for type=loadbalancer services. We are saved in the non restart case because we compare a cached verson of the service to decided if we should even call into the cloudprovider and update resources. This is happening because our reconciliation routines are confused about session affintiy and lbip, leading them to recreate, since inplace update isn't allowed. The impact is at best a few seconds of radio silence, at worst who knows? Repro: Create a loadbalanced service, eg: Wait for it to acquire a vip. Try it out Ssh into master and restart kube-controller-manager: Watch the wgets, you'll notice radio silence after a bit. This is because the targetpool and forwarding rule get recreted, for 2 reasons: affinity mismatch So this check fails: Lbip mismatch: So this check fails: (lbip is retrieved here: )
the session affinity fix is trivial. The easiest change for the lb ip fix is to not actually check the ip, but simply backoff as long as the forwarding rule exists. This is "bad" because of the way we allocate a static-ip, assign it to the fwd rule, then de-allocate it. The alternative is to use to reconcile, but that's racy.
cherry pick


Question: if !found { return nil, fmt.Errorf("'name' is a required parameter.") } <del> port, err := strconv.Atoi(params["port"]) </del> <ins> portString, found := params["port"] if !found { return nil, fmt.Errorf("'port' is a required parameter.") } port, err := strconv.Atoi(portString) </ins> if err != nil { return nil, err }

expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" get se hostnames -o yaml apiVersion: v1beta1 containerPort: "9376" ... obviously, endpoints controller is not happy.


Question: Obj() inUseClaim = st.FromResourceClaim(pendingImmediateClaim). Allocation(&resourcev1alpha2.AllocationResult{}). <del> ReservedFor(resourcev1alpha2.ResourceClaimConsumerReference{UID: types.UID(podUID)}). </del> <ins> ReservedFor(resourcev1alpha2.ResourceClaimConsumerReference{Resource: "pods", Name: podName, UID: types.UID(podUID)}). </ins> Obj() allocatedClaim = st.FromResourceClaim(pendingDelayedClaim). OwnerReference(podName, podUID, podKind).

Apply this patch: Then run: This fails with: Test should pass. See above. If there are N objects in claims.Items, then this loop for _, claim := range claims.Items { objects = append(objects, &claim) } is appending the same pointer N times, pointing at the final item in the list. Adding fixes that mistake, but then it makes the test fail. Something else about the test is buggy, but the bug was being masked by this bug. In the repro case I to the loop a few lines down, just for good measure, but that one is not causing a problem. The line by itself causes the test failure. HEAD N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /priority important-soon /cc Thanks for finding this and the detailed report
/assign
Can you help to fix this?
If we don't here from soonish, then I'll fix it myself.
Sorry, I've been busy with an office accident this week, if you want fix it, please take it
/assign
/unassign


Question: } scaleTest := &HPAScaleTest{ initPods:                    1, <del> totalInitialCPUUsage:        250, perPodCPURequest:            500, </del> <ins> totalInitialCPUUsage:        500, perPodCPURequest:            1000, </ins> targetCPUUtilizationPercent: 20, minPods:                     1, maxPods:                     5, firstScale:                  3, firstScaleStasis:            stasis, <del> cpuBurst:                    700, </del> <ins> cpuBurst:                    1400, </ins> secondScale:                 5, } scaleTest.run(name, kind, rc, f)

Test: https://k8s- Example: https://k8s- The HPA tests for gce-serial are flaking rather regularly; they fail about 2/3 of the time for the last week, and are often the cause of a failed test run. Can we de-flake these? Over multiple test runs the problem seems to be the number of replicas jumping up to 4 unexpectedly: Is the limit not getting set correctly here? Is this an actual bug? /sig autoscaling /priority important-soon /kind failing-test /kind flake
cc:
Recent failures fall into three categories: not starting (, , , , , , ). They fail with error message: increasing unexpectedly (, , ). They fail with error message: waiting for scale down (). This one failed with error message: I suspect 2. and 3. may be caused by test using correct amount of CPU when averaged over minute-long window but spiky (for example using 100% cpu for some time then 0% cpu for rest of the minute). I saw similar behavior when validating recent changes manually. This would result in HPA: behaving as the test expects with before recent changes (because it ignored any spikes during scale*forbidden windows, longer sample windows masking the spike), HPA (correctly) increasing size if spikes align after those changes (failures in group 2.), HPA (correctly) not decreasing size if spikes align after those changes (failure in group 3.), I'll take a look at code of those tests and try to responsible for most of the flakes but I don't have any ideas for a cause yet.
/remove-kind failing-test
I was busy this week but I think I'll be able to prepare a fix for this tomorrow.
/milestone v1.13 /kind failing-test
this tests is failing consistently in
On Friday I verified that CPU usage generated by resource consumer stays really close to the target value but oscillates slightly. To fix this I: Increase CPU usage in the test (resource consumer is implemented in a way that makes me think that deviation fro target are of a fixed size so higher target would make deviation a smaller percent of the target). Lower generated load (the test right now is targeting border between recommendation of 3 instances and 4 instances, I will change the load to something between 2 and 3 instances).

I'm checking if this helps the test.
It seems to have helped. I've sent a PR with those changes for review.
Re-opening until we have at least 3 green runs: https://k8s- /reopen
Reopening this issue. <detailsIn response to : Re-opening because we just had another flake last night: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
But it doesn't work on test grid, the test now fails with: I'll look into this.
Yes, and it's now causing another test to fail as well: https://k8s- Escalating to critical-urgent. This test has been failing for 3 weeks now, and is potentially blocking visibility into other failures. /priority critical-urgent /remove-priority important-soon
This test had a couple of successful runs yesterday but is now flaking/failing in master-blocking: https://k8s- It seems to be passing in master-upgrade. Any chance we could prioritize work on it? It's preventing us from seeing whether fixes for other test failures are working.
There was only 1 flake since merged on Thursday (and ~15 green runs). was an unexpected scale up happening during stabilization. I'll check if lowering CPU utilization a bit lower deflakes the test more but I think it's ok to close this issue now.
Lowering CPU utilization didn't help.
Test jobs where these tests have flaked or failed: https://k8s- (today) https://k8s- (last 3 days ago) https://k8s- (yesterday) https://k8s- (yesterday) https://k8s- (today) https://k8s- (yesterday) ... basically, everywhere the tests run, they're flaky as recently as today. More on Triage: You can see here that across all jobs the HPA tests are flaky, with a "worst flake" perioud of 10/23-10/25, but by no means cleared up.
SIG-Autoscaling's boards are also showing flakiness: https://k8s-, although that looks better than the sig-release test boards. Any idea why?

Thanks for the links I've been looking only at one of those dashboards. I've sent that should reduce flakiness of our tests.
Thanks!
The PR merged. Now let's wait and see if it solves the problem.
The test named in the tile of this issue<sup1</supgot better. Last failures at the dashboards you listed: , (before merged), error before taht was on 10-25), , , . So it looks like the test didn't flake in the last ~2 days. Another two tests<sup2</supare flaking , I'll send a PR that should fix it. I think those flakes are result of behavior of HPA intentionally changing<sup3</supso the PR will just relax expectations of the test. <b1</b[sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability <b2</b[sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale down with External Metric with target value from Stackdriver [Feature:CustomMetricsAutoscaling] and [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale down with External Metric with target average value from Stackdriver [Feature:CustomMetricsAutoscaling] <b3</bHPA no longer delays scale ups based on metrics other than CPU. Flakes are when HPA scales deployment to 2 or 3 so test doesn't observe it having size 1 and times out waiting for that size.
I misread titles of custom metric tests as tests for scaling up somehow. Those flaking tests are tests for scaling down and scale downs sometimes don't happen. I'm looking into why.
While this test has gotten better, it's still flaking some of the time: https://k8s-
I looked at "Latest failures" in the two top categories according to (all the other categories have almost no recent flakes). All of the failures log as if they were not affected by : instead of .

Tests results also say they are running version 1.11... or 1.12... (or more rarely 1.9... or 1.10... but I'm ignoring those for now - they are not related to any changes I made). So it looks like I need to cherry pick the PR into 1.11 and 1.12 branches.
... when I look at git branches it looks like my PR is there. There was with logs that looking like affected it. I'll wait to see if there are more failures in test runs including that change. In case there are I'll add some more logging (I have no ideas for cause of the one failure I saw so far).
/assign
GitHub didn't allow me to assign the following users: jbartosik. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any further update on this? I see it turning into more of a failure than a flake and I see another test failing as well in latest runs
We are due to cut Beta for 1.13 today and would like to understand if this should be a blocker or not.
Also are there cherrypick PRs to backport your change to 1.11 and 1.12 branches?
Potentially related:
6 (consecutive) most recent failures of test do not have judging by their logs. That PR would change them from failures (unexpected scale from 3 to 4 instances during stabilisation) to success (expected 3 to 4 instances during stabilisation). One of the 2 recent failures of looks wrong. One failure was scaling up immediately to 5 instances (wrong). The other was scale up to 4 instances (ok, I'll send a PR relaxing that condition tomorrow). I didn't send PRs cherrypicking test changes to 1.12 (because when I checked it looked like the change was there) not to 1.11 (my changes aren't there so old tests should keep working). merged and it ads more logging, which will help me debug this. Failures in the look like the PR relaxing stability requirement should have helped there too.

the , which has the 6 (consecutive) most recent failures , upgrades the master alone to 1.13(master) and and keep the nodes in old version and then run the old test suite. The latest failing runs show that the tests are run from 1.11.5 branch Doesn't this mean your changes need to be backported to 1.11 branch? That is the reason why you dont see you change in those tests logs
FWIW, the same test also flakes in job which upgrade both master and nodes to 1.13 and runs 1.11 tests.
this failure is currently blocking Beta for 1..13. Can you please investigate these and let us know the path forward.
Fixes done by are all test fixes, not feature fixes. So the problem should be fixed once cherry-pick gets to 1.11. That being said - why are we running 1.11 e2e tests against 1.13 cluster? That assumes full compatibility of both API and implementation. We can never remove a feature without breaking this suite (the tests for the feature will still exist on older branches after all). And the tests are often tweaked to the specifics of implementation. Whenever we make any change to autoscaling algorithm tests need to be updated accordingly. Making tests on 1.11 work well with 1.13 risks they will stop working well with 1.11. Shouldn't the upgrade job create 1.11 cluster, upgrade it to 1.13 and run 1.13 e2e tests on that cluster?
I see these 2 pass in the latest run of https://k8s- Over to to close when he sees fit
We have other flakes on that job now, but HPA is no longer flaking. Thanks so much for the dilligence in getting these flakes cleared up, /close
Closing this issue. <detailsIn response to : We have other flakes on that job now, but HPA is no longer flaking. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/client-go/tools/cache" "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/scheduler/internal/queue" <ins> "k8s.io/kubernetes/pkg/scheduler/profile" </ins> ) func (sched *Scheduler) onPvAdd(obj interface{}) {

What happened: An extra scheduling attempt is observed in some scenarios when kube-scheduler already has already assumed a pod and this has entered the binding routine. If, during this time, a pod update is received, the pod is back to the scheduling queue, even if binding succeeds. This is the correct behavior, as scheduling conditions might change. However this updates could just be unrelated status updates, such as updates to pod conditions. We observe an extra scheduling attempt, which is visible in the log line: Which indicates that an attempt was made when the pod was already scheduled. What you expected to happen: No extra scheduling attempts How to reproduce it (as minimally and precisely as possible): This depends on other controllers running on the cluster that might report pod conditions. One such controller is
/sig scheduling


Question: return cpuUsageNanoCores } <ins> func getSystemUUID() (string, error) { result, err := exec.Command("wmic", "csproduct", "get", "UUID").Output() if err != nil { return "", err } fields := strings.Fields(string(result)) if len(fields) != 2 { return "", fmt.Errorf("received unexpected value retrieving vm uuid: %q", string(result)) } return fields[1], nil } </ins> func getPhysicallyInstalledSystemMemoryBytes() (uint64, error) { // We use GlobalMemoryStatusEx instead of GetPhysicallyInstalledSystemMemory // on Windows node for the following reasons:

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The windows kubelet does not set .status.nodeInfo.systemUUID on the node object. What you expected to happen: The windows kubelet sets the systemUUID from How to reproduce it (as minimally and precisely as possible): Start the windows kubelet and check for UUID: Anything else we need to know?: This will affect users running the out-of-tree vsphere CCM since that depends on the systemUUID, unlike the in-tree version. Also, looks like cAdvisor is the component responsible for providing the systemUUID so let me know if this issue needs to be moved to that repo. Environment: Kubernetes version (use ): Verified on 1.13.1 and 1.14 Cloud provider or hardware configuration: None OS (e.g: ): Windows Server 2019
/sig windows /sig vmware
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: expectedNodeInfo *NodeInfo }{ { <del> pod:         makeBasePod(t, nodeName, "non-exist", "0", "0", "", []v1.ContainerPort{{}}, []v1.Volume{}), </del> <ins> pod:         st.MakePod().UID("non-exist").Namespace("node_info_cache_test").Node(nodeName).Obj(), </ins> errExpected: true, expectedNodeInfo: &NodeInfo{ node: &v1.Node{

We have a pod fake function in , but this is not a base util anymore, we have a lot of arguments, we can replace this with here /sig scheduling /kind cleanup /help This is an easy one to get start. Cleaner code for maintenance.
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? /kind cleanup /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind feature
I want to try this one, the problem we solve here is to replace the *v1.Pod with type PodWrapper {}? /assign
Replace with in
you mean replace ` in I check the code and didn't found out that we have makeBasePod in
I have tried it. But there will be error about cycle import, because package import package, where locate.
yes, I've tried it as well. guess I need to wait for kerthcet's response <img width="805" alt="image" src=""
Sorry, forgot to reply on this, I introduced another PR to help solve this import cycle, but waits for other maintainers' response. So generally, if that pr accepted, we can work on this one then. Thanks for the volunteer.
the import cycle is resolved, you can start your work now, thanks!
thanks for the ping!


Question: fi } <ins> # Setup working directory for kubelet. function setup-kubelet-dir { echo "Making /var/lib/kubelet executable for kubelet" mount --bind /var/lib/kubelet /var/lib/kubelet/ mount -B -o remount,exec,suid,dev /var/lib/kubelet } </ins> function reset-motd { # kubelet is installed both on the master and nodes, and the version is easy to parse (unlike kubectl) local -r version="$("${KUBE_HOME}"/bin/kubelet --version=true | cut -f2 -d " ")"

GCI Test Failing: (kubernetes-gci-gce-alpha-features-release-1.4) ([] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working mysql cluster [Feature:PetSet]) https://k8s-
This one, and appear to be failing because of the same reasons. The files being written by the init containers don't have the right permissions perhaps? This is what I observed so far. Looks like it's mounted with noexec for some reason. cc
nice debug
Random tip, came across this looking for another bug. GCI broken emptydir because kubelet's root-dir is noexec. Petset is just a victim:
Seems to be a dup of I have posted a fix. , Prashanth B wrote:


Question: " ## testsuite block <del> if [[ -e "$juDIR/junit-$suite.xml" ]]; then </del> <ins> if [[ -e "$juDIR/junit_$suite.xml" ]]; then </ins> # file exists. first update the failures count <del> failCount=`sed -n "s/.*testsuite.*failures="([0-9]*)".*/1/p" "$juDIR/junit-$suite.xml"` </del> <ins> failCount=`sed -n "s/.*testsuite.*failures="([0-9]*)".*/1/p" "$juDIR/junit_$suite.xml"` </ins> errors=$(($failCount+$errors)) <del> sed -i "0,/failures="$failCount"/ s/failures="$failCount"/failures="$errors"/" "$juDIR/junit-$suite.xml" sed -i "0,/errors="$failCount"/ s/errors="$failCount"/errors="$errors"/" "$juDIR/junit-$suite.xml" </del> <ins> sed -i "0,/failures="$failCount"/ s/failures="$failCount"/failures="$errors"/" "$juDIR/junit_$suite.xml" sed -i "0,/errors="$failCount"/ s/errors="$failCount"/errors="$errors"/" "$juDIR/junit_$suite.xml" </ins> # file exists. Need to append to it. If we remove the testsuite end tag, we can just add it in after. <del> sed -i "s^</testsuite>^^g" $juDIR/junit-$suite.xml ## remove testSuite so we can add it later sed -i "s^</testsuites>^^g" $juDIR/junit-$suite.xml cat <<EOF >> "$juDIR/junit-$suite.xml" </del> <ins> sed -i "s^</testsuite>^^g" $juDIR/junit_$suite.xml ## remove testSuite so we can add it later sed -i "s^</testsuites>^^g" $juDIR/junit_$suite.xml cat <<EOF >> "$juDIR/junit_$suite.xml" </ins> $content </testsuite> </testsuites>

is not picking the cmd test junit output due to file naming issue. The file name should match regex /assign
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig testing


Question: return result, prefilterStatus.AsError() } <del> nodes := nodeLister.ListNodes() if err != nil { return result, err } if len(nodes) == 0 { </del> <ins> numNodes := g.cache.NodeTree().NumNodes() if numNodes == 0 { </ins> return result, ErrNoNodesAvailable }

In : It seems that nodes are only used in the case there are no predicates. This call can be moved inside . Moving it there saves us from creating the node slice unless it is necessary. Checking if there are no nodes (line 202 below) can be done using . /kind cleanup /sig scheduling
I can work on this


Question: "net/http" "os" <del> v1 "k8s.io/api/core/v1" </del> "k8s.io/apimachinery/pkg/util/sets" kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm" "k8s.io/kubernetes/cmd/kubeadm/app/constants"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --https://k8s- kubeadm-init 06-22 https://k8s- all kubeadm jobs are failing against k/k master. is this failing after we merged?
cc /sig cluster-lifecycle /area kubeadm kinder cluster preparation constructs images locally by downloading tarballs and tagging them with: then running it would find the image locally and not pull it from GCR. does not contain pre-release images, so trying to pull these images would fail. i think the problem here is that we broke the default behavior with
ok, the problem is indeed here: empty string "" should be the same as PullIfNotPresent.
should be fixed by
/triage accepted


Question: volSpec.Fstype = "ext4" } <ins> var volumeMode *v1.PersistentVolumeMode if utilfeature.DefaultFeatureGate.Enabled(features.BlockVolume) { volumeMode = v.options.PVC.Spec.VolumeMode if volumeMode != nil && *volumeMode == v1.PersistentVolumeBlock { glog.V(5).Infof("vSphere block volume should not have any FSType") volSpec.Fstype = "" } } </ins> pv := &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name:   v.options.PVName,

FEATURE REQUEST: vSphere volume plugin does not support volume device yet. Need to add volumeMode in PV spec as other plugin did. release 1.12
/sig storage


Question: currentPods       *v1.PodList ) <del> err = wait.Poll(100*time.Millisecond, 30*time.Second, func() (bool, error) { </del> <ins> err = pollTimed(100*time.Millisecond, 60*time.Second, func() (bool, error) { </ins> currentAPIService, _ = aggrclient.ApiregistrationV1beta1().APIServices().Get("v1alpha1.wardle.k8s.io", metav1.GetOptions{}) currentPods, _ = client.CoreV1().Pods(namespace).List(metav1.ListOptions{})

[sig-api-machinery] Aggregator Should be able to support the 1.7 Sample API Server using the current Aggregator More nuanced logs and details can be found at gave up waiting for apiservice wardle to come up successfully Expected error: <errors.errorString | 0xc420131150: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred /priority failing-test /priority important-soon /kind flake /sig-api-machinery /assign can you please help triage this issue?
has made changes that could have affected this recently. made one that'd be even more suspicious but I don't think it has merged yet :) :)
flakes show steady 2-3/day over the past week: which changes are you referring to? the line that's failing isn't doing any discovery, restmapping, etc, at all... just a straight get to the API:
audit log from the test run shows the requests are returning 404s, not 503s
ah, sorry, I misread the log and thought it was a discovery error. Hm.
hmm, there's a second smaller set of failures that are discovery related: looks like those started flaking after which went in on 5/4 looks like some discovery paths had internal retries that could have masked those 404 errors before. since the error that test is encountering is for an unrelated CRD API that is being created/deleted as part of another test, it makes more sense to switch that check to just ensure the extension API group under test is discovered, rather than assert there are no errors looking for other API groups that we know are being dynamically /removed opened to deflake the aggregator e2e check
/assign resolved the discovery issue. now the failure is the one noted in where calls to the extension API are responding with 404 errors
Kubelet log is reporting that the sample-apiserver exited with a return code of 2 and it also shows with a high restart count for the sample-apiserver.

When I extract the sample-apiserver log I'm seeing multiple "panic: Get https://10.0.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.0.0.1:443: getsockopt: connection refused" Both failures today are in ipvs clusters which seems to have extensive networking problems. /assign
We are looking into finding the root cause into the problems with the ipvs clusters. I am pretty confident they were triggered by a bad PR, just need to identify which one.
Initial hypothesis is that PR caused it.
We think should fix the IPVS issues.
I see merged and this test is now passing in both pull-kubernetes-e2e-gce and ci-kubernetes-e2e-gci-gce. Hence closing the issue. Thanks all
/reopen The test is failing for the same error again in the following jobs https://k8s- It is one of the top flake from last week failing 47 jobs.
can you please take a look and triage? Thanks
this flake is now blocking this test from being promoted to Conformance suite in 1.11. Is it something we can investigate and resolve soon (today or tomorrow) to increase changes of getting this into 1.11? If not this test has to wait until 1.12 to be promoted.
all I see in the test run logs is 404 errors coming back from that check. can you see anything useful in the container logs for the aggregated server?
/cc
Update: I have been trying to reproduce this failure on my own test cluster (go run ...) and it has passed 128 times in a row so far. I think this means that the issue must be with either an interaction between e2e tests or an issue with the job running the tests.
Looking through recent results this crops up as a flake in presubmit and CI across many configurations... I don't think how we run the test jobs should be affecting this in the slightest. Not sure what's wrong here, but it looks like a pretty typical flakey test (some race / buggy setup/teardown?) nb that may not trivially map to the same way the tests are run though, and itself is a wrapper over
does the API server logs throw any light on whats happening on CI test cluster.

The teardown for the aggregator test is wrong, it tries to delete the deployment "sample-apiserver" when the deployment is actually named "sample-apiserver-deployment", but I don't think that's the issue. It's been like that since the test was . I'll look for conflicts with other tests.
We should probably go ahead and fix that anyway, it might be stomping on another test... Jenny Buckley <:
The namespace controller polls each of the api groups periodically, and it gets this internal error back from the endpoint a couple of times. Maybe related?
unclear... I opened to capture the state of the APIService, extension server pods, and those pod logs in failures cases
It flaked again since your PR merged https://k8s-
All I'm seeing is
I don't think that's related... I see 10-30 of those same rejections in apiserver logs of successful test runs (for example, search for ). I don't see anything in the test setup that grants the sample apiserver permission to run those checks, so it doesn't seem like that would be intermittent or flaky.
I am re-opening this issue since I still see intermittent failure of this test are your pending PRs ( and ) targeted towards these remaining issues? /reopen
this looks like 1.7 API Server image is causing more flakes other than just this test - The test has become lot less flaky after increasing the timeout. Since the official policy for GKE is 2 previous versions, and 1.11 is almost baked, this is the plan discussed with offline: leave the 1.7 test as it is right now create a similar test to the 1.7, but with 1.9 (implies creating a new sample-sever, uploading image, and creating the test). get that new test PR merged and running let it soak for several days, see if in 1.7 - 1.9 the flakiness (latency or whatever is causing it) went away.

if it did, evaluate removing the 1.7 and using the 1.9 let us know what you think of this approach
I think you misunderstood my comment... that only related to kubectl being used against a 1.7-level kube-apiserver serving discovery docs for the v1 API, not a generic 1.7-level extension API server
unrelatedly, moving the test to 1.9-level code once 1.11 is released seems reasonable to me
Thank you Jordan, that's the current plan! We will actually create a parallel 1.9-level test, to see if between 1.7 and 1.9 the internals of API Server fixed what's generating the flakiness and see next steps from there.
Sorry I didn't read the previous discussion close enough. The idea of creating a 1.9 test makes sense.
Pinging to see what are the next steps. Who is taking point to create a parallel test?
last I chated with someone on his team will build 1.9 API server image and push it to GCR. Once that image is available then can help writing the parallel test. I am not sure if the image step is done.
It's on the works. Buiding the sample-image showed to be more complicated than what it seemed to be. If it becomes a blocker let me know. Aish Sundar <:
I'd like to know where we are on this, because the 1.7 thing has come up in the context of multi-arch image builds
and
Hi Aaron and Aish, sorry for the delay. The process to compile, create and upload the sample API Server was not properly documented / maintained, and that plus more burning things is what made it take so long. We are trying to finally have it back (and documented by end of this week). Will keep you posted.
Image created with all the refactor happened on that area, trying upgrading the test at present from 1.7 to 1.9
Update: Upgraded the test, and now the test is not passing... working on this.

• Failure [38.067 seconds] [sig-api-machinery] Aggregator Should be able to support the 1.9 Sample API Server using the current Aggregator [It] creating a new flunders resource Expected error: <errors.StatusError | 0xc420c90cf0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: " "rest-flunder-" is forbidden: not yet ready to handle request", Reason: "Forbidden", Details: { Name: "rest-flunder-", Group: "", Kind: "flunders", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 403, }, } "rest-flunder-" is forbidden: not yet ready to handle request not to have occurred
not sure if it's related, but I don't think the test currently gives sufficient permissions to the aggregated server to perform delegated authn/authz checks. we might want to look at merging
tried your fix, same result. The image of the sample-apiserver 1.9 based that I've created is uploaded to If someone want's to give it a try, simply by upgrading 1.0 to 1.1 here:
/assign Can you try to figure out why the test is failing w/ the new image?
/cc
Now waiting on
Is there an ETA on ? This test is flaking quite a bit offlate - https://k8s-
upgrades to 1.10. however it requires more rbac permissions than before which is not fully understood. We will either resolve the permission issue or leave it for later investigation. The upgrade to 1.10 should be merged in a couple of days, if not today.
what issue are you seeing? is relevant?
Yes, it is relevant. It does not fix the issue by itself though.
As for should be ready to be merged.
/reopen I would like some manual confirmation this has been addressed
Reopening this issue. <detailsIn response to : /reopen I would like some manual confirmation this has been addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
after some in-depth investigations, it turns out the 1.10 sample-apiserver would need the following rbac permissions (in addition to the system:auth-delegator role). I'm updating the PR to add these. apiVersion: kind: ClusterRole metadata: name: sample-apiserver rules: apiGroups: resources: '*' verbs: list apiGroups: '' resources: namespaces verbs: list
This test is failing again, FYI: https://k8s-
Hi Josh, TL;DR: we are upgrading to 1.10 and that's what people is talking about in the previous comments.
/reopen I would like to see the test pass with the fix before closing this issue out. thanks
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi please notice the test was renamed to: [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator, and has been passing since merged? https://k8s-
Ah cool thanks that explains it :) I was looking at in release-master-blocking and looks like it hasn't picked by the latest test since its 10/13 run. Closing this issue now based on the api-machinery dashboard now. Will reopen if the release job doesnt turn green the next run. Thanks much.
Is it worth removing the copy of this test that lives in ? Or is it still a release-blocking test?
I think the test will automatically pick the new version of the test in its next run (which should be sometime today, 10/20?). As to whether the test should be in the scale-correctness job , it is a ques for and


Question: kind: %s metadata: name: %s <del> finalizers: - test-finalizer </del> unknownField: unknown spec: foo: foo1

After running k8s conformance testing on kubernetes v1.27, we have leftover custom resources of the format that should not exist. For example) Basic test instructions: We use the latest official kubernetes kubekins image: We extract kubetest binary from kubekins latest-master docker image. We run serial conformance tests as follows. We also run parallel conformance tests as follows.
/sig api-machinery
The following script is used to cleanup the leftover CRDs:
/triage accepted /assign /cc
/area conformance
/sig architecture


Question: Name:  "bar", Image: image, Command: []string{ <del> "/bin/sh", "-c", "sleep 10; fallocate -l 10M file; sleep 10000", </del> <ins> "/bin/sh", "-c", "sleep 10; dd if=/dev/zero of=file bs=1M count=10; sleep 10000", </ins> }, Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{

in this test the rootfs is limited to 5M and the pod is evicted by creating a 10M file with; But this fails if the fs doesn't support the fallocate syscall. The K8s e2e must not itself require support for features that K8s does not require. I am unsure why my overlay fs in containers doesn't support fallocate(2) while the fs on the node (ext4) does. I have queried on the slack. I build my Linux kernel myself but I can't find anything related. If the command is changed to; the test works on an fs that doesn't support fallocate(2). It may seem more "primitive" but it doesn't require a feature that K8s doesn't. <detailsNone </detailsNo response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Interesting /Triage accepted


Question: optional k8s.io.apimachinery.pkg.apis.meta.v1.Time time = 1; // String detailing the error encountered during Attach or Detach operation. <del> // This string maybe logged, so it should not contain sensitive </del> <ins> // This string may be logged, so it should not contain sensitive </ins> // information. // +optional optional string message = 2;

This came up as a follow up to Also rather than opening 5 different issues - we are going to use this github issue to track all follow up items mentioned here -
/sig storage
/assign /assign
: PR
and the rest is fixed in


Question: volumeMounts: - name: kube-dns-config mountPath: /etc/k8s/dns/dnsmasq-nanny <ins> securityContext: capabilities: drop: - all add: - NET_BIND_SERVICE - SETGID </ins> - name: sidecar image: k8s.gcr.io/k8s-dns-sidecar:1.14.13 livenessProbe:

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Apply security best practices for both kube-dns and core-dns. Low hanging fruit: [x] Run as non-root & disallow privilege escalation [x] Run with the default seccomp profile (i.e. ) [x] ReadOnlyRootFilesystem [x] Avoid unnecessary HostPath volumes [x] (n/a) More advanced: [ ] Reduce dependencies in the base image () [ ] Custom seccomp profile [ ] Custom AppArmor profile [x] Drop unneeded capabilities[x] Run as non-root & disallow privilege escalation [x] Run with the default seccomp profile (i.e. ) [x] ReadOnlyRootFilesystem [x] Avoid unnecessary HostPath volumes [x] (n/a) More advanced: [ ] Reduce dependencies in the base image () [ ] Custom seccomp profile [ ] Custom AppArmor profile [x] Drop unneeded capabilities This is based on the long term issue and implemented by PR . Why is this needed: Our system pods should run using security best practices, both to enhance cluster security and serve as examples of best practices to users.
/sig auth /area security


Question: } // If both the pod and labels are unchanged, no update is needed <del> if !podChanged && !labelsChanged { </del> <ins> if !podChangedFlag && !labelsChanged { </ins> return }

This test has started failing on our e2e test suite, erroring out with "Failed to get endpoints..." https://k8s- https://k8s- Example failures https://k8s- https://k8s- cc cc
This may be related to our recently-merged endpoints change in . I'll look into it.
I verified that the flakiness introduced was a result of . A fix in should be merging soon. /assign
GitHub didn't allow me to assign the following users: joelsmith. Note that only can be assigned. <detailsIn response to : I verified that the flakiness introduced was a result of . A fix in should be merging soon. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "ingressclass.go", "kube_proxy.go", "loadbalancer.go", <del> "network_policy.go", </del> "network_tiers.go", "networking.go", "networking_perf.go",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Remove the folder in favor of . Why is this needed: With the refactoring from the Netpol validation , the tests from folder are deprecated and duplicated at this point. Action items: tests tests
/area testing /sig network
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /area testing /sig network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thank you ... !
/triage accepted


Question: } } }`) <ins> var structuralSchemaWithEmptyObject = []byte(` { "openAPIV3Schema": { "description": "weird CRD with empty spec, unstructured status. designed to fit test fixtures.", "type": "object", "x-kubernetes-validations": [ { "rule": "[has(self.spec), has(self.status)].exists_one(x, x)" } ], "properties": { "spec": { "type": "object" }, "status": { "type": "object", "additionalProperties": true } } } } `) </ins>

While migrating webhooks to CEL for one of the structs in our CRD api, I noticed that the empty object in CEL is interpreted wrongly. Empty map object seems to be interpreted correctly. In the example below, both map and object are sent empty () in json output. returns true json (captured on the client side) sent to the apiserver CRD schema No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
CEL only sees the JSON sent to the server, and is correctly evaluated as present. The issue is with the client sending the request populating a value in that field. How is the request being constructed? /remove-kind bug /kind support
/cc /triage accepted
Upon investigation, the problem lies there: We requires either or be set.
Thanks for raising this issue! I was debugging offline with and seems like CRD/CR validation didn't handle the situation of unsetting properties with field type specified . It is a corner case in CRD since we normally would ask for properties defined if the field type is (I could not think of use case which having properties unset). But it might ok to allow it through since it won't hurt from CRD perspective. We will fix in CEL values handling which treat this as empty map passed. /assign
GitHub didn't allow me to assign the following users: jiahui. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : Thanks for raising this issue! I was debugging offline with and seems like CRD/CR validation didn't handle the situation of unsetting properties with field type specified 'Object1object`(I could not think of use case which having properties unset). But it might ok to allow it through since it won't hurt from CRD perspective. We will fix in CEL values handling which treat this as empty map passed. /assign </details
also check if the CRD validation allows the or values, and if so, if the CEL code handles those properly as well
FYI: We do not allow in crd structured schema now:


Question: // Updated updates the lastUpdated timestamp. func (hs *proxierHealthServer) Updated() { <ins> hs.oldestPendingQueued.Store(zeroTime) </ins> hs.lastUpdated.Store(hs.clock.Now()) } // QueuedUpdate updates the lastQueued timestamp. func (hs *proxierHealthServer) QueuedUpdate() { <del> hs.lastQueued.Store(hs.clock.Now()) </del> <ins> // Set oldestPendingQueued only if it's currently zero hs.oldestPendingQueued.CompareAndSwap(zeroTime, hs.clock.Now()) </ins> } // Run starts the healthz HTTP server and blocks until it exits.

We upgraded iptables on one of our hosts, and kube-proxy began logging errors about being unable to acquire the . However, we didn't notice this, because kube-proxy's health check () was still passing (and that's generally what we look at as our sign that something is amiss.) As a result, kube-proxy ran for days without being able to update iptables rules, eventually causing an outage for one of our services. We expected that kube-proxy's health check would fail if it was not able to interface with iptables. Our system was running Debian stretch, running kube-proxy in iptables mode with systemd, with iptables version 1.6.0. Given that starting configuration, the repro case is: iptables to 1.6.2. that kube-proxy begins emitting logs saying that it cannot update iptables because another process has the . that kube-proxy's health check endpoint () does not return an error. Example health check output: <details</details<details</details<detailsNone </details<details</details<detailsThe hard way </details<detailsDocker version 19.03.8, build </details<detailsCalico v3.18.4 </details
/sig network
you upgraded iptables just one node ? what is the version of iptables on other nodes ?
does it means that is not able to never get the lock or is just that sometimes is able to get it?
Yeah we upgraded on just one node at first. Like my original bug report says, we started at 1.6.0, so all the other nodes had 1.6.0. it never gets the lock. You have to restart kube-proxy so it can go through the iptables version detection routine again and change its behavior to the 1.6.2 logic branch. That's fine, it's whatever, but the health check doesn't fail even though kube-proxy can't do its job, which is a bug in my opinion. Also yes, we checked to see if some other process had the lock. It didn't. It's something to do with changes in how iptables actually works after the upgrade to 1.6.2, specifically. I believe this launchpad issue is relevant:
the logic of the healthcheck should fail if is not able to process the updates, right?

I this to the description as well, but here is an example response during this failure mode: And here are what the logs look like: Running gives no output.
This is interesting the and values are the same. This is the healthz logic, it will be unhealthy if the difference between both is greater than the health timeout. is it possible to share the entire kube-proxy log ? I want to discard something
to save me some time, could you please tell me what sort of messages you are looking for? I would rather go search splunk for them than to set up a repro case again.
just the first 1000 lines since kube-proxy starts should be enough
The JSON is kind of meaningless these days; if the proxier thinks it's healthy, it will just write out the current time as both and . It's only when it thinks its unhealthy that it writes out the real time. See I'm not sure how this could happen...
I'm afraid I'm not at liberty to give you 1000 lines of logs without some more targeted context. Company policy and whatnot. What I will tell you is I put the proxy up to and reproduced the issue and pulled those logs, and if you tell me the kind of messages you are looking for, I can pull the (redacted) context around them. What I can give you now, even without being directed, is the more verbose version of what I posted before, i.e., what happens around the time that the error gets logged: <detailsFollowed by the whole list of iptables rules. Then </detailsI can also give you the startup arguments: <details</detailsAnd finally, I can give you these lines from the iptables code, the health check getting registered, and a couple other startup items: <details</detailsAny more than this, though, and I'll have to ask you to tell me something to grep for.
First thing to check is that the healthcheck actually works EVER. Given the same timestamp (we should fix that?) it thinks it is healthy.
/assign
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
A situation: when there is a service or endpoint is constantly updated (which will update the ): so there is a high probability that the difference between the and the is less than the , so this will be considered healthy If you reduce ( iptables-sync-period * 2, the defaults value of flag is 30s), you may see response code is 503, but in fact, it is not easy to estimate how much it should be reduced. so As for case 3, I think there are something that can be optimized.
Oh! That's wrong. We don't care about the time of the most-recently queued update, we care about the time of the oldest queued update that was after the last . So rather than having and updating it every time gets called, we should have a variable something like , and can set it to 0, and should only set it to if it was previously 0. In fact, we don't even need . Kube-proxy would be healthy if or .
Thanks I see, I'd fix it, can I take this one?
sure /assign


Question: } func (s *sourceFile) processEvent(e *inotify.Event) error { <ins> // Ignore file start with dots if strings.HasPrefix(e.Name, ".") { glog.V(4).Infof("Ignored pod manifest: %s, because it starts with dots", e.Name) return nil } </ins> var eventType podEventType switch { case (e.Mask & inotify.IN_ISDIR) > 0:

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Repport Kubernetes version (use ): 1.5.1 (Both Server and Client) Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): UBUNTU 16.04 Kernel (e.g. ): 4.4.0-47-generic Install tools: Chef What happened: Kubelet picks up dot files in --pod-manifest-path causing stuff to breaks by replacing existing pods. What you expected to happen: Exclude dot files from being picked up under --pod-manifest-path How to reproduce it (as minimally and precisely as possible): Take any manifest, copy it as and kubelet will deploy that new manifest Anything else do we need to know: From the slack channel it's been identified to be introduce by (by on Slack)
Just sent a simple fix, while still open for discussion.


Question: var err error token := ts.cache.getToken(azureTokenKey) <ins> if token != nil && !token.token.IsExpired() { return token, nil } // retrieve from config if no cache </ins> if token == nil { <del> token, err = ts.retrieveTokenFromCfg() if err != nil { token, err = ts.source.Token() if err != nil { return nil, fmt.Errorf("acquiring a new fresh token: %v", err) } </del> <ins> tokenFromCfg, err := ts.retrieveTokenFromCfg() if err == nil { token = tokenFromCfg </ins> } <ins> } if token != nil { // cache and return if the token is as good // avoids frequent persistor calls </ins> if !token.token.IsExpired() { ts.cache.setToken(azureTokenKey, token) <del> err = ts.storeTokenInCfg(token) if err != nil { return nil, fmt.Errorf("storing the token in configuration: %v", err) } </del> <ins> return token, nil </ins> } <del> } if token.token.IsExpired() { token, err = ts.refreshToken(token) if err != nil { return nil, fmt.Errorf("refreshing the expired token: %v", err) </del> <ins> klog.V(4).Info("Refreshing token.") tokenFromRefresh, err := ts.Refresh(token) switch { case err == nil: token = tokenFromRefresh case autorest.IsTokenRefreshError(err): klog.V(4).Infof("Failed to refresh expired token, proceed to auth: %v", err) // reset token to nil so that the token source will be used to acquire new token = nil default: return nil, fmt.Errorf("unexpected error when refreshing token: %v", err) </ins> } <del> ts.cache.setToken(azureTokenKey, token) err = ts.storeTokenInCfg(token) </del> <ins> } if token == nil { tokenFromSource, err := ts.source.Token() </ins> if err != nil { <del> return nil, fmt.Errorf("storing the refreshed token in configuration: %v", err) </del> <ins> return nil, fmt.Errorf("failed acquiring new token: %v", err) </ins> } <ins> token = tokenFromSource } // sanity check if token == nil { return nil, fmt.Errorf("unable to acquire token") } // corner condition, newly got token is valid but expired if token.token.IsExpired() { return nil, fmt.Errorf("newly acquired token is expired") </ins> } <ins> err = ts.storeTokenInCfg(token) if err != nil { return nil, fmt.Errorf("storing the refreshed token in configuration: %v", err) } ts.cache.setToken(azureTokenKey, token) </ins> return token, nil }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, when AAD login is configured, refresh token timeout would make kubectl print an error, not able to re-login without manually remove the tokens. Expect kubectl to promote user to login again. Why is this needed: Configure refresh-token timeout from Azure is a way to force re-login after inactivity. kubectl doesn't promote re-login makes users harder to apply timeout policy.
/sig azure
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig platform/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure
the root cause of this issue is in this first gets the cached token (result of device code flow) at it's trying to refresh an expired token at and failed. An error is returned. Since the expired token is still in the cache, it will always hit this path until user removes the token from kubeconfig. the fix should be: the token from cache a retry to re-initiate the device code flow in
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig platform/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
I'm working on it.
are you working on it? I will grab it if I dont hear it from you :)
have the code ready will try PR this weekend Best, Ping
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Closing as merged


Question: } eventBroadcaster := record.NewBroadcaster() <del> recorder := eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "route_controller"}) </del> <ins> recorder := eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "route-controller"}) </ins> rc := &RouteController{ routes:           routes,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: kubect create -f to create statefulset, and describe the statefulset, output the event likes <preEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 statefulset Normal SuccessfulCreate create Claim www-web-0 Pod web-0 in StatefulSet web success 1m 1m 1 statefulset Normal SuccessfulCreate create Pod web-0 in StatefulSet web successful </preWhat you expected to happen: the From of event change to 'statefulset-controller' in output, let know the source of event more obvious, likes the event when create job and other resource(replicaset,replication). <preEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 job-controller Normal SuccessfulCreate Created pod: pi-7gd2m </preHow to reproduce it (as minimally and precisely as possible): create a statefulset and describe it to see the event Anything else we need to know?: Environment: Kubernetes version (use ):kubernetes 1.7 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/assign
/area controller-manager
/sig node


Question: "google.golang.org/cloud/compute/metadata" ) <ins> const LOAD_BALANCER_NAME_MAX_LENGTH = 63 </ins> // GCECloud is an implementation of Interface, TCPLoadBalancer and Instances for Google Compute Engine. type GCECloud struct { service          *compute.Service

GCE has a length limit of 63 characters for its load balancer components (target pools and forwarding rules), and AWS has a length limit of 32 characters. Our current load balancer name construction of clustername-namespace-servicename can exceed both limits, especially AWS's. We should use a shorter name construction method less likely to hit these limits.
Thanks for filing this Alex. My suggestion would be to use a simple UID, as per utils.NewUID(). We'll need to store the mapping from {clustername, namespace, servicename} -UID somewhere stable like etcd. Using anything based directly on {cluster, namespace, name}, for example a stable hash of each, has the problem that the aforementioned tuple is not unique over time. This raises the possibility of e.g. the following scenario: creates mycluster/mynamespace/myservicename, which creates external load balancer named stablehash(mycluster,mynamespace,myservicename) deletes mycluster/mynamespace/myservicename, but at the time MyCloudProvider is down, so kubernetes can't delete ELB named stablehash(mycluster,mynamespace,myservicename). Kubernetes keeps trying in the background to delete the ELB. the mean time user creates a new service named the same as her old one mycluster/mynamespace/myservicename. Kubernetes tries to create an associated ELB named stablehash(mycluster,mynamespace,myservicename), but this fails, because the ELB exists already due to 2 above. ... etc. Badness ensues.
Assigning to Chao Xu, as this makes for a good starter bug.
Hmmmm.... can't seem to assign to caesarxuchao yet. Not sure why.
He has to have write privileges on the repo before issues/PRs can be assigned to him.
But we have many contributors with issues assigned to them that are not in the kubernetes-write group (that only has 48 members). I must be missing something. How do I grant him write privileges on the repo?
We do? That's not how I understand it to work: If you want to add him, go .
has been because the load balancer name changes if the service's UID changes (e.g. if it's deleted and recreated).

Think we can change this? A hash of cluster name, namespace, and service name should get around it, but there may be better ideas as well.
Sure. I'm not confident that I can fix this quickly. If it's urgent, maybe you can assign some one more experienced to fix it? One question: I guess the ELB's name should stay unchanged as long as the "zone+namespace+service name" doesn't change, am I right?
I think I'm the guilty party here. I was unaware that when we create a new service with the same name as a previously existing service, that it needed to keep the same load balancer name and IP. This seems broken to me, but I may be missing the larger context. We can quite safely roll back the change while we decide on the best alternative solution. Q , Chao Xu wrote:
I think the real breakage is that we don't have any way to claim an LB-IP at a larger lifespan than a single service. Alex, can you go back to OP and get a sense of what's happening there to trigger this? , Quinton Hoole wrote:
Will do. Even if the name is kept consistent as it was before, with the way everything currently works the IP won't be consistent.
yeah, I'm not positive that the name has to be consistent. What we need is a way to claim the same public IP address, and we already have that (at least for GCE) Assigning to me, and I'll validate the length requirements, make sure public ip assignment works for AWS and close out this issue.
Note there are two separate things here: (1) a purely-name-based construction can exceed length limits given a single service and (2) some form of across-service consistency is apparently needed. (1) is trivially solvable by either always using a hash or alternatively using a two-stage process where you generate a desired name and only hashify if it's longer than the length limits by truncating it and appending a hash of said name.
AWS's load balancer hasn't been merged yet, so there's not much to check out there. I'm going to be reducing the churn of load balancers later today, which should help prevent IP address changes for services that are modified.

I think we need to be able to figure out which Service an LB is attached to from the Service, so that LBs can't get orphaned. regarding being able to claim an LB IP. I don't think that is covered in revamp... , Brendan Burns wrote:
I think that we need to roll back and think this issue through a bit better. See comment on that PR.


Question: GCE_FLAKY_TESTS=( "DaemonRestartsControllersManager" "Daemonssetsshouldsrunsandsstopscomplexsdaemon" <del> "Jobssareslocallysrestarted" </del> "Resourcesusagesofssystemscontainers" "allowssschedulingsofspodssonsasminionsaftersitsrejoinssthescluster" # file: resize_nodes.go, issue: #13258 "podsw/twosRWsPDssbothsmountedstosonescontainer,swritestosPD" # file: pd.go, issue: #15382

Job should run a job to completion when tasks sometimes fail and are locally restarted Expected error: <*errors.errorString | 0xc20807c990: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred
Couple of failures: from
Another failure:
Another failure:
This test was introduced by and in and . It's currently failing at a rate of about 7%. When it passes, it typically does so in 30-90 seconds. Occasionally it times out after 15 minutes. I think that there's a real bug in the system here. The timeout occurs while waiting for the Job to terminate, which it never does.
I'm going to label it as flaky until the bug has been fixed.
Another portion of flakes:
/sub
any progress on this? Is it being actively worked on?
I've been working on this today. Saw 1 e2e failure in 50 runs. Logs suggest the problem is that half of kubelet's brain thinks it started a pod, and the other half of it thinks it didn't. In fact it didn't. So the test times out waiting for the not-started pod to complete.
Update: the kubelet did start all the pods, and they all terminated, and then the job controller somehow didn't notice that they were done until 40+ minutes later.
This is interesting. There were 3 apiserver 500's around the time of the test failure, and no 500's any other time in the 8 hours the cluster was up, and many other test runs occurred. 500s:
The code that is failing is this in at line 271 the gets an error, and then decides the error is a 500.
can you say anything about the above mentioned apiserver internal error which occurs immediately on receiving a watch request?
Look in the relevant kubelet logs to see how they interpreted the error. It's a bit silly that we printed a stack trace but not the error text :( , Eric Tune wrote:
E1022 04:13:24. 3622 ] Failed to watch *api.Pod: Internal error occurred: too old resource version: 171 ()
This is happening at line 269 of Is this an unexpected condition? If not, why is it a 500?
Oh, I thought that was fixed.

It shouldn't be an internal error, you're out of the watch window and need to re-list. , Eric Tune wrote:
OK I reopened . did the kubelets involved re-list soon after getting these 500s? If so, then this is coincidence and not your problem. , Daniel Smith wrote:
In all 3 cases, they relist immediately, but then they don't re-watch for about an hour. That seems wrong, no? Relist at 3:56 and rewatch at 4:53 Relist at 4:07 and rewatch at 5:00 relist at 4:13 and rewatch at 4:55
The time reported is the end of the watch, not the beginning. The last one it seems you actually didn't get the log line for the watch, actually. But the first two look WAI. , Eric Tune wrote:
Oh, duh.
Working on which I suspect may fix this.
Okay, looking at a new test failure I see that there are expected failures, and the test just times out due to backoff by the kubelet. The following things happen: runs and fails (1) runs and fails (2) backs off for 10s runs and fails (3rd failure) backoff 20s runs and fails (4th failure) backoff 40s runs and fails (5th failure) backoff 1m20s runs and fails (6th failure) backoff 2m40s runs and fails (6th failure) kubelet backoff 5m runs and fails (6th failure) backoff 5m timeout now that at least 15m10s has elapsed in backoffs. Failures have a probability of 50%. Chance of getting 8 fails in a row is 1 in 256. Four pods per test run is 4 in 256 or 1 in 64. I see failures about 1 in 50, so that seems about right. So, I need to rewrite the test so that it writes to the emptydir after the first failure, and succeeds after that.
Ugh. My fix for local breaks non-local restarts.
Happened again.


Question: glog.Infof("Version test passed") } <ins> func runSelfLinkTest(c *client.Client) { var svc api.Service err := c.Post().Path("services").Body( &api.Service{ ObjectMeta: api.ObjectMeta{ Name: "selflinktest", Labels: map[string]string{ "name": "selflinktest", }, }, Port: 12345, // This is here because validation requires it. Selector: map[string]string{ "foo": "bar", }, }, ).Do().Into(&svc) if err != nil { glog.Fatalf("Failed creating selflinktest service: %v", err) } err = c.Get().AbsPath(svc.SelfLink).Do().Into(&svc) if err != nil { glog.Fatalf("Failed listing service with supplied self link '%v': %v", svc.SelfLink, err) } var svcList api.ServiceList err = c.Get().Path("services").Do().Into(&svcList) if err != nil { glog.Fatalf("Failed listing services: %v", err) } err = c.Get().AbsPath(svcList.SelfLink).Do().Into(&svcList) if err != nil { glog.Fatalf("Failed listing services with supplied self link '%v': %v", svcList.SelfLink, err) } found := false for i := range svcList.Items { item := &svcList.Items[i] if item.Name != "selflinktest" { continue } found = true err = c.Get().AbsPath(item.SelfLink).Do().Into(&svc) if err != nil { glog.Fatalf("Failed listing service with supplied self link '%v': %v", item.SelfLink, err) } break } if !found { glog.Fatalf("never found selflinktest service") } glog.Infof("Self link test passed") // TODO: Should test PUT at some point, too. } </ins> func runAtomicPutTest(c *client.Client) { var svc api.Service err := c.Post().Path("services").Body(

Had a report of this over email. Filing this issue so we don't forget to investigate.
Can't reproduce.


Question: }, }, { <del> name: "custom IPv6 networking", </del> <ins> name: "custom IPv6 networking for v1.12.0-beta.2", </ins> cfg: &kubeadmapi.ClusterConfiguration{ Networking:        kubeadmapi.Networking{PodSubnet: "2001:db8::/64"}, CertificatesDir:   testCertsDir, <del> KubernetesVersion: "v1.7.0", </del> <ins> KubernetesVersion: "v1.12.0-beta.2", </ins> }, expected: []string{ "kube-controller-manager",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When I use kubeadm to create an old version k8s cluster, the controller manager pod can not work well. The command like this : "kubeadm init --kubernetes-version=v1.11.2" The error of kube controller manager pod like this : What you expected to happen: The controller manager pod can work well. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Install tools: kubeadm Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig contributor-experience /area kubeadm
Please, use when filing kubeadm issues. /sig cluster-lifecycle
OK, thank you for reminding me.


Question: import ( "net/http" <ins> "strings" </ins> "k8s.io/apimachinery/pkg/util/sets" apirequest "k8s.io/apiserver/pkg/endpoints/request" ) <del> // BasicLongRunningRequestCheck returns true if the given request has one of the specified verbs or one of the specified subresources </del> <ins> // BasicLongRunningRequestCheck returns true if the given request has one of the specified verbs or one of the specified subresources, or is a profiler request. </ins> func BasicLongRunningRequestCheck(longRunningVerbs, longRunningSubresources sets.String) apirequest.LongRunningRequestCheck { return func(r *http.Request, requestInfo *apirequest.RequestInfo) bool { if longRunningVerbs.Has(requestInfo.Verb) {

Is this a BUG REPORT or FEATURE REQUEST?: What happened: In the log: What you expected to happen: The timeout should be more relaxed on /debug/pprof endpoints. How to reproduce it (as minimally and precisely as possible):
/assign


Question: Design docs are in [design](design). <ins> API objects are explained at [http://kubernetes.io/third_party/swagger-ui/](http://kubernetes.io/third_party/swagger-ui/). </ins> ## Primary concepts * **Overview** ([overview.md](overview.md)): A brief overview

The schema understood by the Kubernetes configuration file is not described any where. The current practice mostly is copy/paste existing samples (for example ) and edit them. There is a need to define a comprehensive list of elements, and their documentation, so that other examples can leverage them.
Configuration files specify exactly our API objects. See:
This is exactly what I was looking for but not easily googlable, or documented clearly.
Could you please put a link to the swagger UI here?
The only mention in the public docs is here: which links directly to the struct definition on github. It appears that the Swagger docs have been deprecated, and I'm unable to find any documentation for the KubeletConfiguration resource kind in the new docs. It's referred to in at least one place, but is still not actually covered:


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> TAG = 1.7 </del> <ins> TAG = 1.8 </ins> PREFIX = gcr.io/google_containers all: push

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The documentation suggests that "The default pull policy is IfNotPresent" This is not the case when imagePullPolicy is defined under the spec section in a Pod definition without a specific value. Example: This will result in a Pod having imagePullPolicy as Always. Why is this needed: This is needed for clarification of the default behaviour. It is necessary for the appropriate handling of helm charts, in cases that the imagePullPolicy is empty.
/sig docs
The imagePullPolicy default value is .
In this paragraph you can see that In your reference it points out the default action for the kubelet, not the behaviour when imagePullPolicy is declared. Please do have in mind that the imagePullPolicy option for Never is mentioned only in this paragraph in the document.
If an image tag of "latest" is used, the imagePullPolicy defaults to to "Always": If no tag and no digest are specified on the image, an image tag of "latest" is used: The behavior when no image digest or tag is specified should be documented in the API docs for the and fields:
I am interested on helping this issue, but I am new to kubernete repo, it could take me sometime
I found the following which is well documented in "Best Practices for Configuration" It would be expected to include more information on the options in the main link under
I'd like to work on this one.
Very welcome, but yesterday also mentioned that he is following up on this issue, you can communicate with him before starting

cc
This has failed The two failures are different, and both seemingly totally legitimate. The first one is simply a pod never entering running state within the 5 minute timeout. Expected error: <*errors.errorString | 0xc208475150: { s: "gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s", } gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s not to have occurred The second is not being reachable: Dec 7 01:22:47.275: Unable to connect/talk to the internet: Get dial tcp 74.125.202.147:80: i/o timeout The latter could be a result of a missing MASQ rule on the node, or GCP egress problems. I don't think that this is worth trying to debug further in the context of this test for intra-pod communication. I'll check more generally whether we have a current intermittent problems with respect to pods entering running state, or reaching the internet.
I think this test may be causing builds to time out, too. See comments from me in .
For the future, I actually don't think we need the test at all in the networking test. Maybe even move it do a different test altogether.
This is clearly a bug. I'll send a PR shortly: Regarding the ping to , why did you put in in the first place?
hmmm, not sure what you mean by "in the first place" ? I did not write the test...
sorry, I thought that you'd written that part of the test. I will find out who wrote that, figure out why they put it there in the first place, and then either remove it, or rewrite it to work correctly. Clearly that part of the test needs at least some basic retry to work consistently.
ok great, no problem just wasnt sure what you meant. SGTM. Id be okay w/ deleting the ping entirely regardless of who wrote it, its a very old snippet which doesnt really add much to the overall test suite (and in fact is questionable wether it should even be an assertion at all )
somebody put it there for some reason. Better to understand why before just deleting it.

i looked, its actually rob who i work with :) i will ask him in the morning unless you find him first :) (rrati)
Regarding my alleged "clear bug" above, I was completely wrong. There's nothing wring with that code. And unfortunately, when this test goes into a presumed endless loop (see comment above) it does not produce any output (as the output is cached by Ginkgo until the test completes). So it's very difficult to debug. But in summary I agree with this test is probably never completing, and causing tests to time out. Someone will need to isolate this test in a environment where it produces output and can be debugged. I'll keep trying, but need to head home soon, so if anyone wants to pick it up overnight, go ahead.
Smoking gun:
hmm - I'm just wondering why only this particular test is hanging forever (if we agree that's the case). I don' think that some requests are hanging forever in this test, but not in any other test...
Closing in favor of which should fix this. Or at least some way to set client timeout, which and are still figuring out.
It's happening again.
Talked to offline. I am assigning this one to you for triage. Let me know if you need any help.
AFAICT the last failure was just the pod nettest-7h5jj not coming up
Looks like this should be resolved by your pr: too. :-)
Maybe. Though the webserver only deadlocks in the failure case, so the test is probably still flaky...
Looks like this test is now explicitly flaking by failing, as I expected.
Based on the , it took 15 minutes to pull the image. closing this one as dup of
New Failure:
This looks like a race between populating the endpoints list and the /status, /read methods of the service exposed through the master proxy. It looks like the webserver pod is just trying to confirm that it can reach all peers. We can do this synchronously after having observed all expected endpoints in the test (or re-write the test to use the same pattern we do in services e2e).
The test looks really racy to me. I think the following 15 retries prevents it to fail more often, but not completely.

I will submit a pr to enfore webservers to wait till alll endpoints are up before they start to contact each other.
New failure
I am making this P0
It took more than 4 minutes for a nettest pod to come up. The nettest container is waiting for all endpoints to be ready for 2 minutes plus 15 attempts (5 sec interval) to contact its peers. This happens in less than 4 minutes. Hence the first 2 nettest pods finished contacting peers and declared failure.
The node you want is: I see: that's too much silence ...
From the It looks like kubelet is slow to start the container. Due to the lower logging level in the gke, it's hard to guess what could've slowed down the goroutine...


Question: ginkgo.It("should function for pod-Service: http", func() { config := e2enetwork.NewNetworkingTestConfig(f, false, false) ginkgo.By(fmt.Sprintf("dialing(http) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterHTTPPort)) <del> config.DialFromTestContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err := config.DialFromTestContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> ginkgo.By(fmt.Sprintf("dialing(http) %v --> %v:%v (nodeIP)", config.TestContainerPod.Name, config.NodeIP, config.NodeHTTPPort)) <del> config.DialFromTestContainer("http", config.NodeIP, config.NodeHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err = config.DialFromTestContainer("http", config.NodeIP, config.NodeHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> }) ginkgo.It("should function for pod-Service: udp", func() { config := e2enetwork.NewNetworkingTestConfig(f, false, false) ginkgo.By(fmt.Sprintf("dialing(udp) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterUDPPort)) <del> config.DialFromTestContainer("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err := config.DialFromTestContainer("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> ginkgo.By(fmt.Sprintf("dialing(udp) %v --> %v:%v (nodeIP)", config.TestContainerPod.Name, config.NodeIP, config.NodeUDPPort)) <del> config.DialFromTestContainer("udp", config.NodeIP, config.NodeUDPPort, config.MaxTries, 0, config.EndpointHostnames()) </del> <ins> err = config.DialFromTestContainer("udp", config.NodeIP, config.NodeUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> }) // Once basic tests checking for the sctp module not to be loaded are implemented, this

What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: isErrorRetriableWithSideEffectsFn := func(err error) bool { if canRetry := isWatchErrorRetriable(err); canRetry { klog.V(2).Infof("%s: watch-list of %v returned %v - backing off", r.name, r.typeDescription, err) <del> <-r.initConnBackoffManager.Backoff().C() </del> <ins> <-r.backoffManager.Backoff().C() </ins> return true } if isExpiredError(err) || isTooLargeResourceVersionError(err) {

Currently, reflector uses two backoff managers: One is used for initial list call and other for watch call. Imagine following scenaraion when k8s apiserver is under very heavy load: the beginning, reflector spams k8s apiserver (with exponential backoff) with calls until one of them succeeds (due to APF throtling and receiving ) tries to run request and uses separate backoffManager - so again spams k8s apiserver with requests Watch does not succeed during Etcd compaction window (2.5 - 5 minutes usually) at some point it will receive and go back to point 1, spamming k8s apiserver again with List calls (because first backoffManager for List got reseted after 2 minutes of retries for Watch requests) I propose we combine these two backoff managers into one. If we combine these two backoffManagers into one, it will work like this: At the beginning, reflector spams k8s apiserver (still with exponential backoff) with calls until one of them succeeds Tries to make first watch call instantly (as there is no backoff on this path) If it fails, it will makre retries every (30, 60)s If watch request fails all the times during Etcd compaction window it will go back to List call, but already being throttled with (30, 60)s window. /sig scalability
/cc
/assign
I think the proposed solution makes sense to me. Probably before we start, we should to understand why we ended up with two back off managers by looking into PRs touching them. Maybe the code authors had some specific case in mind.
for thoughts too I agree it makes sense, but I don't know the historical context here.
/sig api-machinery
/triage accepted


Question: // TearDownAt simply discards everything in the directory. func (ed *emptyDir) TearDownAt(dir string) error { <ins> // First remove ready dir which created in SetUp func readyDir := ed.getMetaDir() if removeErr := os.RemoveAll(readyDir); removeErr != nil && !os.IsNotExist(removeErr) { return fmt.Errorf("failed to remove ready dir [%s]: %v", readyDir, removeErr) } </ins> if pathExists, pathErr := mount.PathExists(dir); pathErr != nil { return fmt.Errorf("Error checking if path exists: %v", pathErr) } else if !pathExists {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There is a node in my cluster whose disk is full on root partition, I cleared the disk and then rebooted the node to recover it. After the reboot, the kube-proxy on that node got stuck in "ContainerCreating" status and never changes, and all other pods can't run properly since the kubeproxy is not runing. What you expected to happen: Kubeproxy on that node can run normally after the disk clean and reboot How to reproduce it (as minimally and precisely as possible): the disk full on root partition in my env, leave roughly 100k space on root partition would make this issue more reproduciblethis issue is kind of tricky, to reproduce it, one need to populate the disk repeatedly since some disk space would be cleared because of pod evictionthe kube-proxy got deleted and created repeatedly because of pod eviction and daemon-set mechanism, it would stuck in ContainerCreating status if the disk space is not enough for creating volume contents of kube-proxy configMap volume, the first pod error event is like: MountVolume.Setup failed for volume "kube-proxy" space left on device After it got stuck, the following pod error event woud repeat over and over, kube-proxy is trying to mount configMap volume but always fail MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory the disk and reboot the reboot is neccesary to reproduce the issue, without the reboot the pod eviction would kick in to delete kube-proxy pod the kube-proxy woud stuck in ContainerCreating Status forever, restart kubelet has no effects on that, kube-proxy pod need to be mannully deleted to get recover kube-proxy pod can't mount configMap volume, the following error keeps repeating MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory Anything else we need to know?

: After digging into this issue for a quite long time, i think the reason for this issue is: The configMap volume uses emptyDir internally to create the volume directory, after the directory is created, configMap volume failed to populate it with configMap content because the disk is full, then configMap volume calls emptyDir.tearDown() to clear the temp directory. The problem is that emptyDir uses a "ready" file under /var/lib/kubelet/pods/{podId}/plugin/.. to determine whether the directory is created or not, if the ready file is present, then it wouldn't create the directory. The tearDown method only remove the directory, it doen't remove the ready file, so after the first out of disk error, the diretory creation always get skipped, hence no such file or directory error repeat over and over. Environment: Kubernetes version (use ): v.1.11.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): cent os 7 Kernel (e.g. ): Install tools: kubeadm Others:
/sig storage
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If it's any help, I just encountered this issue in kubelet v1.13.2 under Debian Stretch 9.5, after a restart. I was not out of disk space. Deleting the ready files from /var/lib/kubelet/pods/{podId}*/ helped.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen hit this just now with the same conditions as originally described, but without a reboot.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen </details
/reopen
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks ! this helped me get my cluster running after all pods were evicted and clearing space alone didn't fix the issue.


Question: } func (asw *actualStateOfWorld) MarkDeviceAsMounted( <del> volumeName v1.UniqueVolumeName) error { return asw.SetVolumeGloballyMounted(volumeName, true /* globallyMounted */) </del> <ins> volumeName v1.UniqueVolumeName, devicePath string) error { return asw.SetVolumeGloballyMounted(volumeName, true /* globallyMounted */, devicePath) </ins> } func (asw *actualStateOfWorld) MarkDeviceAsUnmounted( volumeName v1.UniqueVolumeName) error { <del> return asw.SetVolumeGloballyMounted(volumeName, false /* globallyMounted */) </del> <ins> return asw.SetVolumeGloballyMounted(volumeName, false /* globallyMounted */, "" /*devicePath*/) </ins> } // addVolume adds the given volume to the cache indicating the specified

We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by


Question: } pod.Spec.Containers = append(pod.Spec.Containers, rwcontainer) <del> f.PodClient().CreateSync(pod) </del> <ins> pod = f.PodClient().CreateSync(pod) ginkgo.By("verifying that pod has the correct nodeSelector") framework.ExpectEqual(pod.Spec.NodeSelector["kubernetes.io/os"], "windows") </ins> cmd := []string{"cmd", "/c", "echo windows-volume-test", ">", filePath}

similar to PR en route
/sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nil, fmt.Errorf("either run this scheduler via km or else --executor-path is required") } else { if strings.Index(s.kmPath, "://") > 0 { <del> // URI could point directly to executable, e.g. hdfs:///km // or else indirectly, e.g. http://acmestorage/tarball.tgz // so we assume that for this case the command will always "km" ci.Uris = append(ci.Uris, &mesos.CommandInfo_URI{Value: proto.String(s.kmPath), Executable: proto.Bool(true)}) ci.Value = proto.String("./km") // TODO(jdef) extract constant </del> <ins> if strings.HasPrefix(s.kmPath, "file://") { // If `kmPath` started with "file://", `km` in agent local path was used. ci.Value = proto.String(strings.TrimPrefix(s.kmPath, "file://")) } else { // URI could point directly to executable, e.g. hdfs:///km // or else indirectly, e.g. http://acmestorage/tarball.tgz // so we assume that for this case the command will always "km" ci.Uris = append(ci.Uris, &mesos.CommandInfo_URI{Value: proto.String(s.kmPath), Executable: proto.Bool(true)}) ci.Value = proto.String("./km") // TODO(jdef) extract constant } </ins> } else if s.kmPath != "" { uri, kmCmd := s.serveFrameworkArtifact(s.kmPath) ci.Uris = append(ci.Uris, &mesos.CommandInfo_URI{Value: proto.String(uri), Executable: proto.Bool(true)})

I'd like to run local in agent host. The parameters is used to set executor, but it did not set 's arguments. The proposal is to provide an argument to set the local path of in agent host. cc
It seems we have a hidden parameter , I'll try it firstly.
any suggestion?
how did work out? if you ran into problems .. what were they?
Just have a try, it seems not work; I'm on sure whether configuration issue.
, just go through the code; the will only point to the binaries in master and Mesos agent will fetch it. It did not match this requirement. Do you have time on this? I'd like to work on this and let you review it :).
, would you help to review the patch?
Yes, just provided some feedback on your PR , Klaus Ma wrote: