Question: }) const ( <del> influxdbService      = "monitoring-influxdb" influxdbDatabaseName = "k8s" podlistQuery         = "show tag values from "cpu/usage" with key = pod_id" nodelistQuery        = "show tag values from "cpu/usage" with key = hostname" sleepBetweenAttempts = 5 * time.Second testTimeout          = 5 * time.Minute </del> <ins> influxdbService       = "monitoring-influxdb" influxdbDatabaseName  = "k8s" podlistQuery          = "show tag values from "cpu/usage" with key = pod_id" nodelistQuery         = "show tag values from "cpu/usage" with key = hostname" sleepBetweenAttempts  = 5 * time.Second testTimeout           = 5 * time.Minute initializationTimeout = 5 * time.Minute </ins> ) var (

The following E2E test is failing in the GCE 1.5-1.6 upgrade jobs. Please take a look and let me know ASAP if this is a 1.6 blocker or not (I think it may be a flake, but need you to confirm). https://k8s- Expected error: <*errors.errorString | 0xc421d5ef10: { s: "expected to find one replica for RC or deployment with label heapster but got 0", } expected to find one replica for RC or deployment with label heapster but got 0 not to have occurred
/assign
can you help get this looked at and triaged ASAP? Thanks.
I pinged by email. He's really the one who should look at this.
The problem seems to be not with the test itself but with Kubernetes or test framework, since the test failed on starting Heapster. I'll investigate more later today.
I haven't got time to investigate. This seems to be unrelated to monitoring itself, so any help during MTV day would be great. If not I'll try to fix this tomorrow
thanks
The problem is in , in function . The function is called immediately from , w/o waiting for system pods (heapster) to start. We should give some time for heapster to become ready, as it takes few minutes after cluster creation.
CC
thanks a lot for investigation! is going to fix it.


Question: } out.MetricName = in.Metric.Name out.Selector = in.Metric.Selector <del> currentAverageValue := *in.Current.AverageValue out.AverageValue = &currentAverageValue </del> <ins> if in.Current.AverageValue != nil { currentAverageValue := *in.Current.AverageValue out.AverageValue = &currentAverageValue } </ins> return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The cluster started giving a lot of NODE DENY error in the log corresponding to restarting kube-apiserver make another one crash in a 3 apiserver setup. (say systemctl restart kube-apiserver on controller-1 will crash controller 2 or 3) It's as if node were show as but were not registered anymore?! What you expected to happen: The cluster to work as it did the weeks before How to reproduce it (as minimally and precisely as possible): I haven't fixed this yet and pod are stock in pending most of the time even after restarting controllers, workers, etc. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: 10 workers with 64GB on 3 controllers VM. OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.5 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.5 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g.

): Linux staging-controller-002 4.4.0-139-generic -Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: ansible Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery
/cc seems that it's the graph in node authz is not ready to hande requests. probably respecting infomer's func can make it initialized gracefully. while the log you provides didn't show why the apiserver's crushed. could you provide more detail?
if this occurred right at server start, it is expected the graph will take a short amount of time to populate
If you tell me what more info you need, I'll be glad to provide you with it, maybe the apiserver flags used? Also, I restart the one crash-dumping, one of the other two will start outputting the same dump. Looking back at the logs, this seems to have started when I moved from 1.10.3 to 1.12.1 (now 1.12.2), I just hadn't realized it until recently as we had a big cluster wide problem yesterday and had to investigate some more. But it is still outputting this same dump non-stop even though it seems to be working at the moment and that API server does respond to kubectl commands without issues. Any specific info you'd like me to get?
from the stack trace, sadly we can't find out which exact line trigger that nil pointer panic. the behavior of apiserver could be unstable on the fresh start (due to rbac authz, crd-apiservice registration, etc). are we planing a graceful start mechanism for the server?
the apiserver endpoint is used to determine if the server is ready to accept traffic, and will not return a 200 response until those startup tasks have completed
/healthz does respond (200), while still crash-dump-looping. EDIT: anything I can test specifically to help differentiate between RBAC authz, crd-apiservice reg, etc?
right, I was saying that is the mechanism for graceful startup. unfortunately, the way golang works, if you propagate a panic (), you lose all stack information from it by default. Some of our panic propagation logs or preserves that, but not all.

I'll open a fix to preserve that information. we can see the request that errored, though: can you see if the same resource (autoscaling/v2beta1/horizontalpodautoscalers) is what causes the panic each time? if so, can you check that resource per namespace to isolate a particular problematic object? those are not related to the panic, as far as I can tell
Still reading, but: Did you skip 1.11? Skipping releases isn't actually supported.
Right now, having been up for a while, it alternates almost perfectly (E114 sometime coming twice in a row) between E114 and I114, basically once per second. These two errors correspond to the one you quoted (HPA) & the other to the HTTP2 panic. The cluster was upgraded from 1.10.3 to 1.12.1 (to 1.12.2) and from what I could gather last week, this started after the upgrade to 1.12.x branch but we didn't realize it until we had to investigate a cluster issue 3 weeks after the upgrade.
Yes, I did, I didn't know that... should I roll back to 1.11 and than back again to 1.12? Or what are the effects and what can be done?
There's at least two things going wrong, one is: the other is whatever is actually crashing. At this point I'm not sure if rolling back to 1.11 would make things worse or better, if your cluster is functioning I probably wouldn't do; if your cluster is nonfunctional and you need it fixed then I might roll the dice. I think we should be able to get a fix for the first thing out soon, but it'll be a while before you'd be able to deploy it unless you're willing to build your own apiserver & run it.
So, any way to diagnose what should have been updated in, I guess, etcd? I mean, is this similar to a schema change that is missing some steps? Thanks!
callstack logging fixed in and
Any chance this will also get merged in 1.12.3 so I could get a meaningful crash dump with the fix?

:)
sure, this will get into 1.12.3 via
So here is the new dump I've got under 1.12.3: Does it make more sense to anyone and is it pointing to the reason for the stack dump?
apologise, it didn't get into 1.12.3. (i thought it would). if you don't feel like waiting until next release, how about building ur own apiserver from release-1.12 HEAD as suggests? (run under this project)
Oh well, I can wait for 1.12.4... (Building it didn't work out and I'd have to build a custom docker, etc. I'll wait for the next release...)
We have the same issue and we reproduced with new kubernetes api-server Stacktrace:
thx for providing the clue! fixed via


Question: return describePersistentVolume(pv, events) } <ins> func printVolumeNodeAffinity(w PrefixWriter, affinity *api.VolumeNodeAffinity) { w.Write(LEVEL_0, "Node Affinity:t") if affinity == nil || affinity.Required == nil { w.WriteLine("<none>") return } w.WriteLine("") if affinity.Required != nil { w.Write(LEVEL_1, "Required Terms:t") if len(affinity.Required.NodeSelectorTerms) == 0 { w.WriteLine("<none>") } else { w.WriteLine("") for i, term := range affinity.Required.NodeSelectorTerms { printNodeSelectorTermsMultilineWithIndent(w, LEVEL_2, fmt.Sprintf("Term %v", i), "t", term.MatchExpressions) } } } } // printLabelsMultiline prints multiple labels with a user-defined alignment. func printNodeSelectorTermsMultilineWithIndent(w PrefixWriter, indentLevel int, title, innerIndent string, reqs []api.NodeSelectorRequirement) { w.Write(indentLevel, "%s:%s", title, innerIndent) if len(reqs) == 0 { w.WriteLine("<none>") return } for i, req := range reqs { if i != 0 { w.Write(indentLevel, "%s", innerIndent) } exprStr := fmt.Sprintf("%s %s", req.Key, strings.ToLower(string(req.Operator))) if len(req.Values) > 0 { exprStr = fmt.Sprintf("%s [%s]", exprStr, strings.Join(req.Values, ", ")) } w.Write(LEVEL_0, "%sn", exprStr) } } </ins> func describePersistentVolume(pv *api.PersistentVolume, events *api.EventList) (string, error) { return tabbedString(func(out io.Writer) error { w := NewPrefixWriter(out)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: Now that it's a field, it needs a separate entry in 'kubectl describe'


Question: b.Fatalf("op %d: invalid op %v", opIndex, concreteOp) } } <del> if err := waitUntilPodsScheduled(ctx, podInformer, b.Name(), nil, numPodsScheduledPerNamespace); err != nil { // Any pending pods must be scheduled before this test can be considered to // be complete. b.Fatal(err) } </del> <ins> // Some tests have unschedulable pods. Do not add an implicit barrier at the // end as we do not want to wait for them. </ins> return dataItems }

is a very useful option in scheduler perf test framework to define an operation without waiting for it to finish. However, the current testing logic still waits for all the pods in this operation to be schedulable, in the last step: the number for Pods in this operation to be schedulable This can be problematic as some case just want to generate Pods (for example, to test internal queuing or sorting behavior/latency). The following test command times out as those large-cpu Pods are expected to not fit on any node: So, given is commented as: I'm proposing to relax the semantics to not wait for the Pods to be schedulable if the operation is specified as . /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign WDYT?


Question: len(nodes.Items)) } <del> ips := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) </del> <ins> var family v1.IPFamily if framework.TestContext.ClusterIsIPv6() { family = v1.IPv6Protocol } else { family = v1.IPv4Protocol } ips := e2enode.GetAddressesByTypeAndFamily(&nodes.Items[0], v1.NodeInternalIP, family) </ins> clientNodeInfo = nodeInfo{ name:   nodes.Items[0].Name, nodeIP: ips[0], } <ins> ips = e2enode.GetAddressesByTypeAndFamily(&nodes.Items[1], v1.NodeInternalIP, family) </ins> serverNodeInfo = nodeInfo{ name:   nodes.Items[1].Name, <del> nodeIP: ips[1], </del> <ins> nodeIP: ips[0], </ins> } })

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-network-kind-ipvs,dual,master sig-network-kind-ipvs,IPv6,master Which test(s) are failing: [x] KubeProxy should set TCP CLOSE_WAIT timeout [x] Conntrack should be able to preserve UDP traffic when server pod cycles for a NodePort Service [x] KubeProxy should resorve connection reset issue (tracking in ) Since when has it been failing: Oct 28 Testgrid link: sig-network-kind-ipvs,dual,master  Reason for failure: Both appear to be failing on missing conntrack entries ex. Anything else we need to know:
/sig network /cc /assign
I'm spinning up a kind cluster with ipv6+ipvs to see if I can replicate this test locally
/triage accepted
heh, the job was not running it, it started to run when we changed the tag I meant this test and the PR where we renamed the kube-proxy test
this one has an issue already open, and I have a PR for fixing it
reached out to me on Slack and we identified as the high priority. I have tested this locally and it fails on both IPTables and IPVS kind clusters locally. I'm digging through the cluster logs now.
I' ve logging to conntrack with level v4 recently just for these problems, try to bump your kube-proxy logging to v4
seems that there are less test failures on the IPv6 jobs, let's keep watching them :eyes: However, the dual-stack jobs are legit failures, basically because on dualstack clusters, the node has 2 internal IPs, and those tests use the internal IP for asserting some operations, and the problem is that they are picking the internalIP of the wrong IP family, hence the test fails The solution should be to add the InernalIp correspoding to the cluster primary IP familty, you can obtain it with but it seems that we have to modify the function or add a new one that returns the nodes and the corresponding ips
Hey Thanks for all the help, will look into the areas of the code and try to bump the logging.


Question: "k8s.io/kubernetes/pkg/controller/util/node", "k8s.io/kubernetes/pkg/controller/volume/persistentvolume/util", "k8s.io/kubernetes/pkg/controller/volume/scheduling", <ins> "k8s.io/kubernetes/pkg/credentialprovider", "k8s.io/kubernetes/pkg/credentialprovider/aws", "k8s.io/kubernetes/pkg/credentialprovider/azure", "k8s.io/kubernetes/pkg/credentialprovider/gcp", "k8s.io/kubernetes/pkg/credentialprovider/secrets", </ins> "k8s.io/kubernetes/pkg/features", "k8s.io/kubernetes/pkg/fieldpath", "k8s.io/kubernetes/pkg/kubectl",

What happened: There is to manage import ways under but that doesn't work at our upstream CI. We can enable it by But if doing that, we will face the following error with What you expected to happen: I hope the check is always running to block invalid import ways at upstream CI system. How to reproduce it (as minimally and precisely as possible): As line into arguments.InputDirs of
/sig testing /area e2e-test-framework


Question: fi fi # Command <del> kubectl annotate -f "${file}" annotated=true "${kube_flags[@]}" </del> <ins> # Command # We need to set --overwrite, because otherwise, if the first attempt to run "kubectl annotate" # fails on some, but not all, of the resources, retries will fail because it tries to modify # existing annotations. kubectl-with-retry annotate -f $file annotated=true --overwrite "${kube_flags[@]}" </ins> # Post-condition: mock service (and mock2) and mock rc (and mock2) are annotated if [ "$has_svc" = true ]; then kube::test::get_object_assert 'services mock' "{{${annotations_field}.annotated}}" 'true'

shippable flake: It seems that this test needs to retry on update failures. cc who fixed some other similar test flakes recently and might want to take a swing at this one.
cc
We discussed about it in Assigning to
Thanks . Removing team/control-plane, as team/CSI seem to have this in hand.
Aha, my PR was also hit by this flake just now. I'll try to fix this tmr.
has the same problem.


Question: }) } } <ins> func TestSortPodIPs(t *testing.T) { testcases := []struct { name        string nodeIP      string podIPs      []string expectedIPs []string }{ { name:        "Simple", nodeIP:      "", podIPs:      []string{"10.0.0.1"}, expectedIPs: []string{"10.0.0.1"}, }, { name:        "Dual-stack", nodeIP:      "", podIPs:      []string{"10.0.0.1", "fd01::1234"}, expectedIPs: []string{"10.0.0.1", "fd01::1234"}, }, { name:        "Dual-stack with explicit node IP", nodeIP:      "192.168.1.1", podIPs:      []string{"10.0.0.1", "fd01::1234"}, expectedIPs: []string{"10.0.0.1", "fd01::1234"}, }, { name:        "Dual-stack with CRI returning wrong family first", nodeIP:      "", podIPs:      []string{"fd01::1234", "10.0.0.1"}, expectedIPs: []string{"10.0.0.1", "fd01::1234"}, }, { name:        "Dual-stack with explicit node IP with CRI returning wrong family first", nodeIP:      "192.168.1.1", podIPs:      []string{"fd01::1234", "10.0.0.1"}, expectedIPs: []string{"10.0.0.1", "fd01::1234"}, }, { name:        "Dual-stack with IPv6 node IP", nodeIP:      "fd00::5678", podIPs:      []string{"10.0.0.1", "fd01::1234"}, expectedIPs: []string{"fd01::1234", "10.0.0.1"}, }, { name:        "Dual-stack with IPv6 node IP, other CRI order", nodeIP:      "fd00::5678", podIPs:      []string{"fd01::1234", "10.0.0.1"}, expectedIPs: []string{"fd01::1234", "10.0.0.1"}, }, { name:        "No Pod IP matching Node IP", nodeIP:      "fd00::5678", podIPs:      []string{"10.0.0.1"}, expectedIPs: []string{"10.0.0.1"}, }, { name:        "No Pod IP matching (unspecified) Node IP", nodeIP:      "", podIPs:      []string{"fd01::1234"}, expectedIPs: []string{"fd01::1234"}, }, { name:        "Multiple IPv4 IPs", nodeIP:      "", podIPs:      []string{"10.0.0.1", "10.0.0.2", "10.0.0.3"}, expectedIPs: []string{"10.0.0.1"}, }, { name:        "Multiple Dual-Stack IPs", nodeIP:      "", podIPs:      []string{"10.0.0.1", "10.0.0.2", "fd01::1234", "10.0.0.3", "fd01::5678"}, expectedIPs: []string{"10.0.0.1", "fd01::1234"}, }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kl := testKubelet.kubelet if tc.nodeIP != "" { kl.nodeIPs = []net.IP{net.ParseIP(tc.nodeIP)} } podIPs := kl.sortPodIPs(tc.podIPs) if !reflect.DeepEqual(podIPs, tc.expectedIPs) { t.Fatalf("Expected PodIPs %#v, got %#v", tc.expectedIPs, podIPs) } }) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In a dual-stack cluster with IPv4 as default family (first kubelet IP), a container is created with a env var set to . When the container is running, properly contains the IPv4 But at runtime, this env var (used in container args) is expanded to the pod IPv6 instead of the IPv4. When using , it should always return the value actually exposed as and not something else. Kubernetes version:Cloud provider or hardware configuration: Bare metal cluster OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux 5.8.0-59-generic 66~20.04.1-Ubuntu SMP Thu Jun 17 11:14:10 UTC 2021 x8664 x8664 x86_64 GNU/Linux Network plugin and version (if this is a network-related bug): cilium 1.10 Others: CRI-O 1.21 + crun.
Note: I have a configuration that matches the ones that had IP ordering issue before was fixed. So it may be related.
/sig network
Just a guess, but I wonder if it may be caused by
That's it. The fix from properly sort IP when computing the Pod status, but the sorted IPs are then replaced by computed IPs as this log shows.
/triage accepted Thanks for the detailed report
/assign


Question: }) } <ins> postTimeoutCh := make(chan struct{}) </ins> ts := httptest.NewServer( withDeadline( WithTimeout(

Unit test in package flakes there is a race condition between the go routine that executes the request handler and the timeout go routine that waits for the handler to finish /sig api-machinery
/assign
I have a PR in the works
should fix it, I can easily reproduce it with the PR, the flake is not reproducible any more cc
/triage accepted


Question: klog.Errorf("detach azure disk: disk %s not found, diskURI: %s", diskName, diskURI) } <del> newVM := compute.VirtualMachine{ Location: vm.Location, </del> <ins> newVM := compute.VirtualMachineUpdate{ </ins> VirtualMachineProperties: &compute.VirtualMachineProperties{ HardwareProfile: vm.HardwareProfile, StorageProfile: &compute.StorageProfile{

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Instead of using the VM CreateOrUpdate () function of the Azure Go SDK, use the VM Update function () in the azureclient () Why is this needed: Currently, the Azure Cloud Provider is using the VM CreateOrUpdate call when e.g. detach/attaching disks to a VM. In some scenarios, this will re-create a VM that was deleted by an external application. This can lead to orphaned VMs or blocked resources. The Update function will only update an existing VM and will fail if the VM does not exist.
/sig azure
/assign
/sig cloud-provider
Could you describe more about the scenarios? Have you noticed such issues? I wonder adding etags to the API call could help to prevent this?
If you have an application that maintains your K8s clusters, e.g. takes care of OS update/patching. If the application wants to patch the OS of a worker node, it would drain the node first. If draining times out, it will go ahead and delete the VM anyway. But K8s might still be detaching disks. These detach operations, because they use CreateOrUpdate, will recreate the VM. Is there a reason why you would want to create a VM i.e. why CreateOrUpdate is required and Update would not work? AFAIK, Azure Compute RP does not support etags.
that sounds reasonable, thanks change to use function would be safer since it's only detach/attach disk operation. I will do that.
thanks for the information. Reasonable enough. /assign
/milestone v1.16
BTW, the reason why we only use func in the beginning is that Azure Compute go sdk did not provide a func at that time...
Great. Thanks!


Question: withClaimVolumeMode(&modeFile, newClaimArray("claim14-12", "uid14-12", "10Gi", "volume14-12", v1.ClaimBound, nil, annBoundByController, annBindCompleted)), noevents, noerrors, testSyncClaim, }, <ins> { // syncVolume output warning when pv is prebound to pvc with mismatching volumeMode "14-13 - output warning when pv is prebound to pvc with different volumeModes", withVolumeVolumeMode(&modeFile, newVolumeArray("volume14-13", "10Gi", "uid14-13", "claim14-13", v1.VolumePending, v1.PersistentVolumeReclaimRetain, classEmpty, annBoundByController)), withVolumeVolumeMode(&modeFile, newVolumeArray("volume14-13", "10Gi", "uid14-13", "claim14-13", v1.VolumePending, v1.PersistentVolumeReclaimRetain, classEmpty, annBoundByController)), withClaimVolumeMode(&modeBlock, newClaimArray("claim14-13", "uid14-13", "10Gi", "", v1.ClaimPending, nil)), withClaimVolumeMode(&modeBlock, newClaimArray("claim14-13", "uid14-13", "10Gi", "", v1.ClaimPending, nil)), []string{"Warning VolumeMismatch"}, noerrors, testSyncVolume, }, { // syncVolume output warning when pv is prebound to pvc with mismatching volumeMode "14-13-1 - output warning when pv is prebound to pvc with different volumeModes", withVolumeVolumeMode(&modeBlock, newVolumeArray("volume14-13-1", "10Gi", "uid14-13-1", "claim14-13-1", v1.VolumePending, v1.PersistentVolumeReclaimRetain, classEmpty, annBoundByController)), withVolumeVolumeMode(&modeBlock, newVolumeArray("volume14-13-1", "10Gi", "uid14-13-1", "claim14-13-1", v1.VolumePending, v1.PersistentVolumeReclaimRetain, classEmpty, annBoundByController)), withClaimVolumeMode(&modeFile, newClaimArray("claim14-13-1", "uid14-13-1", "10Gi", "", v1.ClaimPending, nil)), withClaimVolumeMode(&modeFile, newClaimArray("claim14-13-1", "uid14-13-1", "10Gi", "", v1.ClaimPending, nil)), []string{"Warning VolumeMismatch"}, noerrors, testSyncVolume, }, { // syncVolume waits for synClaim without warning when pv is prebound to pvc with matching volumeMode block "14-14 - wait for synClaim without warning when pv is prebound to pvc with matching volumeModes block", withVolumeVolumeMode(&modeBlock, newVolumeArray("volume14-14", "10Gi", "uid14-14", "claim14-14", v1.VolumePending, v1.PersistentVolumeReclaimRetain, classEmpty, annBoundByController)), withVolumeVolumeMode(&modeBlock, newVolumeArray("volume14-14", "10Gi", "uid14-14", "claim14-14", v1.VolumePending, v1.PersistentVolumeReclaimRetain, classEmpty, annBoundByController)), withClaimVolumeMode(&modeBlock, newClaimArray("claim14-14", "uid14-14", "10Gi", "", v1.ClaimPending, nil)), withClaimVolumeMode(&modeBlock, newClaimArray("claim14-14", "uid14-14", "10Gi", "", v1.ClaimPending, nil)), noevents, noerrors, testSyncVolume, }, { // syncVolume waits for synClaim without warning when pv is prebound to pvc with matching volumeMode file "14-14-1 - wait for synClaim without warning when pv is prebound to pvc with matching volumeModes file", withVolumeVolumeMode(&modeFile, newVolumeArray("volume14-14-1", "10Gi", "uid14-14-1", "claim14-14-1", v1.VolumePending, v1.PersistentVolumeReclaimRetain, classEmpty, annBoundByController)), withVolumeVolumeMode(&modeFile, newVolumeArray("volume14-14-1", "10Gi", "uid14-14-1", "claim14-14-1", v1.VolumePending, v1.PersistentVolumeReclaimRetain, classEmpty, annBoundByController)), withClaimVolumeMode(&modeFile, newClaimArray("claim14-14-1", "uid14-14-1", "10Gi", "", v1.ClaimPending, nil)), withClaimVolumeMode(&modeFile, newClaimArray("claim14-14-1", "uid14-14-1", "10Gi", "", v1.ClaimPending, nil)), noevents, noerrors, testSyncVolume, }, </ins> } err := utilfeature.DefaultFeatureGate.Set("BlockVolume=true")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If a user specifies volumeMode: Block in creating PVC and neither plug-in nor external provisioner doesn't support block, a PV is created, but is not bound to the PVC because the PV and PVC don't match. So, PVC will stay pending forever without any error messages. What you expected to happen: In above case, a proper error message is shown to users to make them understand what's wrong. How to reproduce it (as minimally and precisely as possible): up plug-in and external provisioner that don't support block. PVC using volumeMode: Block. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: KVM OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: hack/local-up- Others: /sig storage


Question: } } <del> select { case <-execFuncDone: case <-time.After(10 * time.Millisecond): t.Fatalf("%d: timed out waiting for execFunc to complete", i) } </del> <ins> <-execFuncDone </ins> } }

this is you
Hmm, I had upped waiting from 100ms to 500ms and that seemed to fix it. Do you have any better ideas?
Don't wait? :)
If the only reason you would timeout would be the code is broken, that's why the test runs have -timeout set.
Then the test fails immediately. It's waiting for the net.Listen calls to finish so it can connect to the specified local ports.
*finish enough so that socket connections work
So I can remove the timeout gating in my tests?
Yes ----- Original Message -----
Just failed again: https://travis-
Seems to be failing frequently now.
If this is test is failing again, my gut says it's not the same root cause. The original failure was a timeout issue. This new one is connection reset by peer. I've got the test running in a loop waiting for a failure but haven't seen one yet. is this still failing frequently?
No. It had failed twice in ~10 builds when I reopened this issue, but hasn't failed since. Reclosing. Will file a new issue if it recurs.


Question: dns = append([]string{kl.clusterDNS.String()}, hostDNS...) } if kl.clusterDomain != "" { <del> nsDomain := fmt.Sprintf("%s.%s", pod.Namespace, kl.clusterDomain) dnsSearch = append([]string{nsDomain, kl.clusterDomain}, hostSearch...) </del> <ins> nsSvcDomain := fmt.Sprintf("%s.svc.%s", pod.Namespace, kl.clusterDomain) svcDomain := fmt.Sprintf("svc.%s", kl.clusterDomain) dnsSearch = append([]string{nsSvcDomain, svcDomain, kl.clusterDomain}, hostSearch...) </ins> } return dns, dnsSearch, nil }

Needed for new DNS schema transition. was reverted in because of Need to add 'metatdata' to e2e names-to-resolve list for GCE
OK, so we have a real problem. if libc resolver has a limit of 6 search paths, and we intend to prepend at least 3 ( ) there's a very real possibility that a random machine on which someone wants to run kube will have more than 3 search domains. Even worse, what we really want is to prepend 4 (otherwise we have to do a release with which you MUST update your DNS server), and even GCE has 3 search paths. So the question is how to handle it? 1) Use less search paths ourselves. Force users to use more specific names or even FQDNs. Mitigates but does not eliminate the problem. 2) Log an error in kubelet, but soldier on: "some of your DNS search domains may get dropped". It really is "may" since we don't know what every resolver lib does. Need a way for people to fix this, e.g. a kubelet flag --dns-search-paths="," which trumps 3) Make kubelet hard-stop. Log an event, set a Condition, refuse to run. Still need a way to fix this as in (2). for consult.
I suggest the following strategy to move forward: checks for the number of search paths in host's and mark the node unhealthy if it has more than a maximum number of search paths. We can also add a kubelet flag to let users provide custom search paths for pods. being backwards compatible while introducing the new "svc" sub-domain.
For reference RHEL recommends running dnsmasq on systems affected by this, because dnsmasq isn't affected - links to some discussion about the ABI on it.
Could we drop '' for now from the search path? I think i'm less concerned about that one if it's unlikely people change it.
I like it. If we drop for now, we can keep compat and squeek by on GCE. We still need to solve the problem overall.

How about the following: kube dns pod to be upgraded as part of the next release (v0.18.0) kube dns create dns records with and without sub-domain (This is the current plan) kubelet to replace existing search paths with and in the next release (v0.19). Existing pods will not face any resolving issues as long as they don't use FQDNs or have their namespace set to . Pipelining the changes provides some time for users to update their pods if they use FQDNs. Does this work for Openshift , Tim Hockin wrote:
We already dropped the old paths for support, so what you've proposed is fine for us. ----- Original Message -----
raised a good point - Clayton's suggestion doesn't work until search paths are transitioned. We need to be sure: a) running pods can resolve "svc" and "" to "" b) new pods can resolve "svc" and "" to "" or "". depending on search paths Then we can transition search paths to So I think Vish is right, but minor nits: kube-dns release that supports both old and new names. users to upgrade kube-dns after 0.18 before 0.19. only new search paths in 0.19; running pods are fine, new pods are fine : Make kube-dns drop support for old names This will let us squeek by on GCE. In the mean time, we can consider how to make kubelet warn when it detects a problem, and how to mitigate said problem.
/cc
re: dnsmasq, I played with it and got it doing what we need wrt nameservers but NOT wrt search paths. As far as I can tell, it expects to pass queries on unmolested, and that the client has proper search paths. Is there some config I am not seeing that makes it apply search paths to incoming queries before forwarding them? Otherwise we're still stuck at 6.
Is the instruction I got internally. ----- Original Message -----
dnsmasq manpage specifically says it ignores everything but 'nameserver' directives from any file. That means it does not do search path expansion, which is exactly what I observe. DNS forwarding works but it only forwards the queries, not expand them.


Question: # Metadata for driving the build lives here. META_DIR := .make <del> # Our build flags. # TODO(thockin): it would be nice to just use the native flags.  Can we EOL #                these "wrapper" flags? KUBE_GOFLAGS := $(GOFLAGS) KUBE_GOLDFLAGS := $(GOLDFLAGS) KUBE_GOGCFLAGS = $(GOGCFLAGS) </del> <ins> ifdef KUBE_GOFLAGS $(info KUBE_GOFLAGS is now deprecated. Please use GOFLAGS instead.) ifndef GOFLAGS GOFLAGS := $(KUBE_GOFLAGS) unexport KUBE_GOFLAGS else $(error Both KUBE_GOFLAGS and GOFLAGS are set. Please use just GOFLAGS) endif endif </ins> # Extra options for the release or quick-release options: KUBE_RELEASE_RUN_TESTS := $(KUBE_RELEASE_RUN_TESTS)

We should just use GOFLAGS
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig contributor-experience
/assign


Question: Follow:         true, Previous:       true, SinceSeconds:   &sinceSeconds, <ins> TailLines:      &tailLines, </ins> SinceTime:      nil, // test a nil custom marshaller with omitempty NonPointerTime: sinceTime, }, <del> expected: url.Values{"container": {"mycontainer"}, "follow": {"true"}, "previous": {"true"}, "sinceSeconds": {"123"}, "emptyTime": {""}, "nonPointerTime": {"2000-01-01T12:34:56Z"}}, </del> <ins> expected: url.Values{"container": {"mycontainer"}, "follow": {"true"}, "previous": {"true"}, "sinceSeconds": {"123"}, "tailLines": {"0"}, "emptyTime": {""}, "nonPointerTime": {"2000-01-01T12:34:56Z"}}, </ins> }, }

How works: : prints all lines. : prints the last line. : prints no lines. (Usually used with to follow new logs) How works: : prints all lines. : prints the last line. : prints all lines. <-------------- Diffierent. The reason is that the is ignored when converting the parameters Confirmed with What is the expected behavior? /cc
I've also noticed that even if or some arbitrarily low number is specified, it tries to read the whole logfile into memory. If it's a big file it can crash the worker node.
Do you mean the docker implementation? If I read their code correctly, docker read the whole log before tail into memory once tail is specified, which may cause problem if tail is a very large number. So in our own implementation, I've avoided that. ;) However, where do you see the ?
Tail should be *int in the API, is it not? On Oct 21, 2016, at 6:01 PM, Lantao Liu wrote: How docker logs works: docker logs --tail=-1 CONTAINERID: prints all lines. docker logs --tail=1 CONTAINERID: prints the last line. docker logs --tail=0 CONTAINERID: prints no lines. (Usually used with -f to follow new logs) How kubectl logs works: kubectl logs --tail=-1 PODNAME: prints all lines. kubectl logs --tail=1 PODNAME: prints the last line. kubectl logs --tail=0 PODNAME: prints all lines. <-------------- Diffierent. The reason is that the tail=0 is ignored when converting the parameters . Confirmed with /cc — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Agree tail=0 no logs is intended behavior On Oct 21, 2016, at 6:01 PM, Lantao Liu wrote: How docker logs works: docker logs --tail=-1 CONTAINERID: prints all lines. docker logs --tail=1 CONTAINERID: prints the last line. docker logs --tail=0 CONTAINERID: prints no lines. (Usually used with -f to follow new logs) How kubectl logs works: kubectl logs --tail=-1 PODNAME: prints all lines.

kubectl logs --tail=1 PODNAME: prints the last line. kubectl logs --tail=0 PODNAME: prints all lines. <-------------- Diffierent. The reason is that the tail=0 is ignored when converting the parameters . Confirmed with /cc — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Yeah, tail is . But if , kubelet will see .
I think the linked to is the problem.
The obvious workaround is to instead use: kubectl logs --tail=1
/sig cli
The line pasted by chao implies omitempty should not be false for tail.
I hate to bump, but 'bump'. Still an issue here. Can confirm works but does not.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale This is still relevant.
Any good news?


Question: } if strings.HasPrefix(fi.Name(), ".") { <del> klog.Errorf("Ignoring file: %s", fi.Name()) </del> <ins> klog.V(5).Infof("Ignoring file (starts with '.'): %s", fi.Name()) </ins> return nil } if !fi.IsDir() { <ins> if fi.Mode()&os.ModeSocket == 0 { klog.V(5).Infof("Ignoring non socket file %s", fi.Name()) return nil } </ins> return w.handlePluginRegistration(event.Name) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is more of an enhancement. What happened: The Plugin Watcher is setup to trigger on file activities when they occur in the subdirectory that it scans. During startup, the Watcher is passed (from within the Kubelet) the Kubelet's root plugin directory as a location to scan for newly plugins - Because the kubelet's plugins subdirectory is also used by existing in-tree drivers (including csi) as a location to store plugins artifacts, the Plugin Watcher is getting triggered by directory/file activities occurring in that subdirectory that has nothing to do with plugin registration as shown in the output below: What you expected to happen: The Plugin Watcher should not be getting triggered by file/directory activities not related to plugin registration/unregistration. One possibility is to have the Watcher scan one directory below plugins like or something similar How to reproduce it (as minimally and precisely as possible): Run plugin watcher with a CSI driver. cc
/sig node
I think you should only place the plugin socket file in the plugin registration directory but not the plugin data.
agree with we can only place the registration file in the specified path instead of placing all data into it. On the other hand, you can place data into the /var/lib/kubelet/plugins/registration even if registration created, that's a no-end loop.
As I mentioned in issue, existing in-tree volume plugins were already using that directory as a location to store storage operation artifacts (not just csi). Here is a shortened list of driver locations that use /var/lib/kubelet/plugins in their code already: And several more locations. The fix would be to have plugin watcher scan a subdirectory of plugins (say ) instead of plugins directly as it is doing today. The watcher functions as intended. Except now it picks up all directory activities whether related or not to plugin registration.
Well given the fact that the pluginwatcher is beta this is a bit of an issue. My understanding of the "beta" guarantees is that we want to be backwards compatible for all plugins published with the beta API. Meaning that even if we move the registration directory to a new location, we'll need to keep scanning the old directory (to pick up v1beta plugins).

From a 10 000 feet point of view it seems easier to me if we move the in-tree volume plugin's data. Though it might be a lot of work. Another less painful option, but a bit grey is the following: Enforce plugins to end with this suffix is recommended for device plugins (though not enforced) and I believe is a convention on linux. This is a bit grey because we would still be breaking v1beta users but since we're moving a convention from "recommended" to "enforced" it seems "less" breaking to me (and I haven't seen device plugins not using that extension). What do you think ?
I like the suffix idea. It can be a recommendation (which csi is already doing).
Hmm, I've just deployed a new k8s cluster on 1.12.0 and noticed this repeating in the logs on a few of our pods. Just for my own sanity, is this something I should be worried about or can I ignore it until a fix is released?
you can ignore these errors. if it's a recommendation then we'll keep spamming the logs with all the data files. Another thing to keep in mind, it seems like CSI plugins puts a lot of data (and creates a lot of directories). It might be worth considering limiting the recursion to one level. I'm concerned we might exhaust the amount of fds by tracking the plugins data.
yeah CSI keeps track of volume operation metadata in the plugins dir. This probably will not change. I understand this is beta. However, besides CSI and Device plugin, are there any other implementers out there for this feature ? Would it be possible to propose a new location (i.e. <pluginsDir/reg/?
let me know if you open a PR for this as discussed.
thanks for open the issue with detail logs, i can see two problems by looking at logs and code. 1) plugin watcher trying to dial non-socket files. 2) too many logs generated (CREATE and DELETE). I a suggestion to fix of this issue we can fix those problems by: 1) check file mode when handle CREATE event, only attempt to dial file when file mode is socket.

2) increase log verbose level for both 'handleDeleteEvent()' and 'handleCreateEvent()' plugin watcher still has activity when any file create/delete under the directory, but they become light weighted (CREATE file will only trigger file mode check, DELETE file event will trigger plugin searching) i am not sure we want to go further to limit directory depth where plugin can live. it has two drawbacks IMO: 1) it is hard to enforce, not sure everyone will read code/readme. 2) it provides less flexibility to plugins: if want to organize many plugin socket files with directory.
These are definitely good suggestions! It seems like there might be some confusion as to why we are thinking of doing this: Most plugins use the plugin directory to store data (actually a lot of files / dirs) Watching all the directories and files created inside the plugin directory puts more burden on the Kubelet in terms of cpu / memory consumption Watching all the directories and files created inside the plugin directory quickly raises the number of fd consumed By limiting the search of the plugins to one directory, we would be severely limiting the ability of the module to unintentionally harm the overall Kubelet. It seems actually pretty easy to enforce, as it's a simple condition. As for not everyone reading code / readme, we can add the documentation to the gRPC interface if this is a concern. In terms of flexibility, is there a real need to create a plugin in a lower directory? Is there a plugin that has so many sockets that a single directory underneath the plugin directory isn't enough? Naively it seems to me that is enough for the different plugins we enable.
ping on the above :)
Naively it seems to me that $ is enough for the different plugins we enable. i can not speak from plugin developer, but both below looks reasonable to me , or as long as we can communicate to plugin developers clearly and effectively (gRPC interface documentation is needed), the directory depth restriction is reasonable to me.
I like . However, why not put plugintype one level up as . For instance
Is the suggestion to hard code the number of levels that the recursion algorithm will search to 3? Or just expect a socket only at a hard coded level (e.g. socket must be nested 3 deep)? Copying my comment from for consistency: Another option I can think of: recursively scanning but before scanning a directory check if it is mounted ().

This will be backwards compatible and will prevent scanning of volume data Thoughts?
i like this idea better if it is always the case that: volume data is mounted, we can modify the watcher code to not watch the mounted directory in the first place.
Skipping mounted dir along with restructuring search path to known (shallow) dir sounds like a good plan. Though with the new suggested paths, sockets will be placed at a location where mounts should not exist.
I think that would be a safe assumption. The plugin directory is only used as a target path for global mounts.
Ephemeral volume types may not need to mount, it could just be a symlink or plain directory. Can we assume any ephemeral volume CSI driver won't implement global mount?
Yes, in-tree ephemeral volumes today don't do a global mount so the mounts directory is which is not the plugin dir. So no issue there. For CSI the will be the global mount path (this plugin dir). Emphermal volumes shouldn't implement call -- but nothing prevents them from doing so. Maybe let's start with just checking for mount and ignoring mount directories. If it is still an issue we can consider adding a configurable black list of paths. Thoughts?
That sounds good. Also, putting the socket files in a specific location that does not clash with mounts will also help (i.e. )
/milestone v1.13 /priority important-soon
I just want to point out that one of the issue we have with recursively watching all directories is that it will significantly increase the amount of fd consumed. Given that plugins actually store data there and in a dynamic way my fear is that it will exhaust the available fds for no real good reasons. Limiting the depth sounds like a reasonable solution to that issue.
btw, because of this verbosity in my cluster writes 5G logs every day and filled up my rootfs (which is small). Is there any way to decrease this verbosity?
We can do both. Don't search mounted dirs, and max depth of search. Who is working on this?
That's pretty bad, can you open another issue to track that.
we have a PR in flight discussing this issue (recursive scanning) here: Thinking more about this, the fact that too much logs is generated is clearly linked to this specific issue and how we process plugins.

In any case I'll be fixing that issues too
how confident are we in getting this fixed by Code freeze tomorrow? 5pm PST
I expect things to wind up today, changes in the above PR aren't very contentious and don't involve significant changes


Question: } type ServiceController struct { <del> cloud       cloudprovider.Interface kubeClient  client.Interface clusterName string balancer    cloudprovider.TCPLoadBalancer zone        cloudprovider.Zone cache       *serviceCache </del> <ins> cloud            cloudprovider.Interface kubeClient       client.Interface clusterName      string balancer         cloudprovider.TCPLoadBalancer zone             cloudprovider.Zone cache            *serviceCache eventBroadcaster record.EventBroadcaster eventRecorder    record.EventRecorder </ins> } // New returns a new service controller to keep cloud provider service resources // (like external load balancers) in sync with the registry. func New(cloud cloudprovider.Interface, kubeClient client.Interface, clusterName string) *ServiceController { <ins> broadcaster := record.NewBroadcaster() broadcaster.StartRecordingToSink(kubeClient.Events("")) recorder := broadcaster.NewRecorder(api.EventSource{Component: "service-controller"}) </ins> return &ServiceController{ <del> cloud:       cloud, kubeClient:  kubeClient, clusterName: clusterName, cache:       &serviceCache{serviceMap: make(map[string]*cachedService)}, </del> <ins> cloud:            cloud, kubeClient:       kubeClient, clusterName:      clusterName, cache:            &serviceCache{serviceMap: make(map[string]*cachedService)}, eventBroadcaster: broadcaster, eventRecorder:    recorder, </ins> } }

If a LoadBalancer cannot be created (e.g. quota limit hit), there is currently no way to know why other than reviewing the logs (?). We could put an status message / object into LoadBalancerStatus.
We could make an event for this too Justin Santa Barbara < :
+1 for event
I think that is reasonable, but we need to be careful how we define it. Event also sounds good. , Justin Santa Barbara < :
I'm in favor of status as well as an event. The pattern I recommend is an array of conditions, similar to NodeCondition, where each contains at least a Status, Reason, and Message.
+1 for an event. I'd argue this is important enough for usability that we should get it into v1.
How often does this failure mode occur?
I unfortunately don't have stats, but it can happen for a handful of reasons. I think these are the most likely: Cloud provider unavailability/errors Ran out of quota from the cloud provider The cluster is using a cloud provider that doesn't support external load balancers I expect the third one is most common by far. We could perhaps move that into the API object validation rather than writing events to handle that particularly case, but using events would solve all the cases together.


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: We face But there is no way to generate default config file How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version 1.9.2
I'll take a look it /sig scheduling
/assign
Hi, same issue with controller-manager
Any updates here? , and don't have this flag. Is it even will be supported?
/cc
if not set, component should be use default value; why we'd like to dump all default config?
Because the documentation on the config file format is either nonexistent or difficult to find. If the program itself can generate a template, that fulfills (most of) the need for documentation of the format.
+1 for this, from my expirence, add an option to write the default config is really helpful.
/assign
also in 1.10
Deprecating all flags, prompting folks to use config files in advance is great, except when there's no documentation nor example anywhere. That means that we can't actually migrate to the new settings, and I can't seem to find clear alpha/beta/stable status for the new config file spec, nor timeline for the removal. It is also pretty confusing that the scheduler expects us to use a config file, but the controller-manager does not?
This issue indeed applies to kube-controller-manager. And there should already had deprecation policy.
According to , the flag will be available in . Since I can't find any reference online to an example config file, perhaps one could be provided or linked to here?
will you be able to pick my patch and generate a config file? I do not get my devbox handy so can only post here until Monday.


Question: framework.Skipf("Environment does not support getting controller-manager metrics - skipping") } <ins> ginkgo.By("Geting default storageclass") defaultClass, err := c.StorageV1().StorageClasses().Get(defaultScName, metav1.GetOptions{}) framework.ExpectNoError(err, "Error getting default storageclass: %v", err) pluginName := defaultClass.Provisioner </ins> controllerMetrics, err := metricsGrabber.GrabFromControllerManager() framework.ExpectNoError(err, "Error getting c-m metrics : %v", err) <del> storageOpMetrics := getControllerStorageMetrics(controllerMetrics) ginkgo.By("Creating an invalid storageclass") defaultClass, err := c.StorageV1().StorageClasses().Get(defaultScName, metav1.GetOptions{}) framework.ExpectNoError(err, "Error getting default storageclass: %v", err) </del> <ins> storageOpMetrics := getControllerStorageMetrics(controllerMetrics, pluginName) </ins> invalidSc = &storagev1.StorageClass{ ObjectMeta: metav1.ObjectMeta{

Which jobs are flaking: () () Which test(s) are flaking: Reason for failure: Timeout: Anything else we need to know: This test flaking about 10% of the time in this job. /sig storage /kind flake /priority important-soon cc
/remove-kind failing-test
/assign
I noticed one issue with the test is that it's not filtering metrics by plugin name. So it will just count the last plugin's metric. I'm not sure if the metrics collected are in any sorted order or not. If not, then when you have a suite that runs multiple plugins, then you might be pulling from the wrong metric.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
still very prevalent in sig-release dashboards
I have pushed which should hopefully fix this.
Also PR against 1.17 branch -


Question: go gc.Run(5, stopCh) defer close(stopCh) <ins> // this makes sure the garbage collector will have added the pod to its // dependency graph before handling the foreground deletion of the rc. timeout := make(chan struct{}) go func() { select { case <-time.After(5 * time.Second): close(timeout) } }() if !cache.WaitForCacheSync(timeout, gc.HasSynced) { t.Fatalf("failed to wait for garbage collector to be synced") } </ins> err = rcClient.Delete(toBeDeletedRCName, getForegroundOptions()) if err != nil { t.Fatalf("Failed to delete the rc: %v", err)

https://k8s- Failed: TestBlockingOwnerRefDoesBlock {garbagecollector}
cc:
https://k8s- Failed: TestBlockingOwnerRefDoesBlock {garbagecollector}
I don't have a good theory yet. It's occurring , so I'll fix the more urgent issues first.
If its that low, its not a blocker. Moving to 1.6.1
Could this be a race condition? Perhaps needs some retry logic? Note that the creation never reported an error that would be fatal to the test:
I checked the log, it appeared when GC was processing the foreground deletion of the owner RC, GC didn't find any dependents of RC. This probably is the same race we expected in no.1. To fix the test, the test can wait some seconds before sending the deletion request, so that the garbage collector will have time to update its dependency graph.


Question: return err } <del> // Share processes </del> <ins> // Record flags that the user explicitly changed from their defaults o.attachChanged = cmd.Flags().Changed("attach") </ins> o.shareProcessedChanged = cmd.Flags().Changed("share-processes") return nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using without specifying should modify existing container images without trying to create a new one. Instead it tries to create a new container and returns an error that isn't set: What you expected to happen: This command should change all container images to instead of returning an error. /sig cli /kind bug /priority important-soon
/assign /triage accepted
My kubernetes cluster version is 1.19.0,and i use kubectl-debug version 0.1.1 try to debug my pod,bug i got error: <img width="1470" alt="image" src="https://user-"Who one know about that


Question: export CGO_ENABLED=1 export CC=${KUBE_LINUX_AMD64_CC:-x86_64-linux-gnu-gcc} ;; <ins> "linux/386") export CGO_ENABLED=1 export CC=${KUBE_LINUX_386_CC:-i686-linux-gnu-gcc} ;; </ins> "linux/arm") export CGO_ENABLED=1 export CC=${KUBE_LINUX_ARM_CC:-arm-linux-gnueabihf-gcc}

Currently is dynamically linked but is not built as a position independent executable. It would be great if it could be built with Go’s flag to enable ASLR. I’m not a security expert, and I’m not familiar with those concepts, but it seems that ASLR is a pretty common (and recommended) security mechanism. If I understand correctly, statically linked binaries (e.g. all the Kubernetes binaries except ) do not require/support PIE so this would only apply to . There was an attempt at using Go’s in but it seems that it was abandoned due to an incompatibility issue with Windows.
/sig release
cc
ASLR stands for Address Space Layout Randomization ref:
conditionally building only Linux binaries in PIE for ASLR seems acceptable. the problem is k8s is using a bash library for building which is not very flexible. also not clear how this would work for bazel builds.
I know at least some distributions which build go binaries per default with . do you see any reason to not turn it on for Linux per default?
i think it should be turned on. my comment was that our build infra is a bit entangled at places. /priority backlog the label seems appropriate.
Whoa! timely - https://go- also https://go-
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I will work on this issue.
/sig security
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle stale
/remove-lifecycle stale
there are outstanding questions about this PTAL, I can lend some pointers to the build but I'd like more input on if this is desirable / worthwhile. Some notes: buildmode=pie does not seem to be supported for all platforms even on linux, which seems problematic given that we compile for many platforms. Custom builds can of course set this without the Kubernetes project choosing to do so, see .
IMO, there's value in building as PIE on the platforms where that is possible, but this is a low priority / nice-to-have. shows one way that building a PIE to enable ASLR can make attacks harder even though Go is memory-safe. It could also be a mitigation against a hypothetical memory-unsafety bug in the Go runtime.
not sure if it moves the discussion forward but ASLR is one mechanism to address NIST 800-53 control requirements for memory integrity/protection (ie SI-16). CIS mapped this in one of their . Note that is an older mapping and rev 5 is coming soon which keep this in as a requirement. I'm an expert enough not to call myself an expert in anything but I can argue it both ways but ultimately it comes down to this - we do a lot in all layers of a system for security and sometimes those help compliance, similarly we do a lot in all layers of a system for compliance that in no way help security but pay the bills. You can judge which is the greater good (or evil).
ROP mitigation is probably the most valuable aspect of this since Kubelet is always running in memory. Its binary may be a rich source of gadgets, of which their offsets in the binary need to be well known to an attacker.
cc to take a look
Started to follow-up on this in


Question: } else if !volMounted || cache.IsRemountRequiredError(err) { // Volume is not mounted, or is already mounted, but requires remounting remountingLogStr := "" <del> if cache.IsRemountRequiredError(err) { </del> <ins> isRemount := cache.IsRemountRequiredError(err) if isRemount { </ins> remountingLogStr = "Volume is already mounted to pod, but remount was requested." } glog.V(12).Infof(volumeToMount.GenerateMsgDetailed("Starting operationExecutor.MountVolume", remountingLogStr)) err := rc.operationExecutor.MountVolume( rc.waitForAttachTimeout, volumeToMount.VolumeToMount, <del> rc.actualStateOfWorld) </del> <ins> rc.actualStateOfWorld, isRemount) </ins> if err != nil && !nestedpendingoperations.IsAlreadyExists(err) && !exponentialbackoff.IsExponentialBackoff(err) {

TL;DR: kubelet keeps trying to remount secrets and configmaps into pods over and over again. My environment: kubernetes 1.3.0 running on Ubuntu 14.04. I used the instructions here to set things up: (i.e. etcd, flanneld and the kube* processes are running on the host OS, not in containers or docker-in-a-vm ala minikube) In trying to track down high disk write rates, I found that the partition I use for my etcd WAL log was getting about 40kb/s write traffic. etcd metrics show that I'm getting 3 qps get and 3qps getrecursive. Write traffic to etcd is essentially nil (pods are all stable). This is a single node system with ~30 user pods, most of which are doing nothing unless I sent traffic to them (which I'm not while doing these measurements). In the upstart kubelet log, I see this over and over again: So it's repeatedly remounting already-mounted secret volumes and configmaps into pods. Analyzing the unique pod IDs, this appears to be affecting all of them: I suspect the 2 missing pods are from old deployments. This goes on ad infinitum every 3-5 seconds, hitting 2-3 pods each time, so presumably some kind of maint loop. The code branch in is this one: and that err is populated from this line: But beyond that I'm not sure why it is consistenly thinking that the mount operation didn't stick. Any ideas where to go from here?
Could it be as simple as that RequiresRemount method in , and are while all the other plugins ?
I can recreate this problem on master with the following sample: Leave it running for 10 mins or so and shows:
Looks like this is by design : The plugin(s) are telling the system to keep remounting to update the contents of the volume. Patching the following to stops the repeated re-mount. Folks are probably using this mechanism to keep the data mounted to be up-to-date. So not sure we should "fix" this behavior. Thoughts please?
Yes, current design keep remounting to update the contents because currently there is no direct way of informing the changes to kubelet.

The remounting normally happens every 10~30 seconds from my observation. In case this behavior cause performance issue, we can consider optimize it.
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow.
Sure. We will work on that. Thanks for posting the issue. Jing 2016721Steffen Hanikel
I'm facing the same issue. Kubernetes 1.3.2 on GKE Container stuck on the ContainerCreating phase since volumes never get mounted. Since default secret keep being remounted all the time, my secret never get mounted:
Hi Taras, I think your problem is different. The log shows that /var/lib/kubelet/pods/ -53de-11e6-b2f2- no such file or directory Looks like it very much related to a name collision bug and there is fix being reviewed Jing , Taras wrote:
Hi Yes, probably it related. Waiting for the fix to get to release to test it on GKE. I needed to downgrade nodes to the 1.2.5 to be able to run pods.
Hello, a parameter to say if the secret need to be updated and remounted periodically would be great :) For most of my pods i just need a secret during the pod first start, and i don't necessarily want the content to stay an be remounted over and over in the pod.
Thanks for your suggestions. I think the log might be a a little bit misleading. If the secret volume is already mounted, it won't mounted ago, But it does rewrite the secret. I think we can avoid rewriting by comparing the contents. Is that good enough? Thanks! Jing , osixia wrote:
yes, how i use secrets not rewriting the content would be great, to avoid rewriting deleted files especially.
do you mean you have some cases that you want to delete some files under secrete directory, and don't want the system to rewrite those files after deletion? It would be great if you could let us know more details about your use case so that we are more clean about what needs to be done. Thanks!

Jing , osixia wrote:
i don't know if it's revelant and may be i just do stupid things just let me know :) a litle example: I have an openldap image, when the container starts for the first time if no database is provided it creates a new one with an admin password. When the database is created the admin password is stored encrypted and i don't want it to stay in the container environment in clear text. (i do the same for mariadb images) So as i don't want the admin password to stay in clear text in the container environment i can't set it via environment variable. I found a solution with secrets : I set the admin password in a configuration file, and put that file in a secret. When a new pod is created the secret was mounted in a volume and the container takes the admin password from the configuration file, create the database and delete the configuration file from the secret's mounted volume. I can start several pods this way with the same secret and admin password and don't have the admin password remaining in the container environment or file system in clear text.
Thank you for posting your use case. We plan to add a feature for addressing it, track it under . Please let me know if you have any suggestions. Thanks! Jing
thank you very mutch :) I have an other thought about the service-account default token, i guess this one will be automatically remounted, but is there any way to not mount this secret to a pod ? I'm concerned about security issues if a pod is compromised, that will allow someone to request the kubernetes api ?
For the problem about excessive disk first raised by I think maybe kubelet should remember which secret UID was last written, and not rewrite if the secret UID has not changed. What do you think For If you want to not have the token mounted, you can do this: It isn't pretty, but it works.
thanks but the trick seems to not work anymore :( returns: if i remove the pod keep crashing with :
What do the pod logs say for the crashing container (e.g. kubectl logs i-have-no-sa)?
The logs are empty
I think your pod is failing because of the combination of image/command.

This gist works: For example, this shows that 's workaround works:
it worked, thanks a lot :)
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow. Sure. We will work on that. Thanks for posting the issue. We're running 1.4.7 now and still seeing a high volume of these messages: With about 125 Pods running in our cluster I'm getting about 6000 log messages per hour similiar to this.
I have similar issue to with fresh kubeadm install on kubernetes 1.5.1
I'm facing the same issue only on one node of my cluster, it fills the number of mounts pretty quick leading to unresponsive OS ( 32k mount )
could you provide some more details about your case? Do you use nfs or gluster in the pods? Could you also give a some of the mounts you see on the node so we can have a better understanding what might cause the issue? Thanks!

Sure sorry, i had lots of tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f259-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-bf6f-11e6-b5ec- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-ddaa-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 The pod -f25f-11e6-8d1a- was restarting a lot ( a misconfiguration on my part ) i'm using version 1.4.7
which version of kuberentes you are using? I see the exact mounts you posted are repeated many time. In the current mount code path, it should check whether the path is already mounted and if it is yes, it will skip to mount it again. I will check again our code path.
Same here with 1.5.2 on CoreOS - it wasn't the problem with 1.3.7, same setup.
Same issue with 1.5.2_2 on CoreOS latest stable as well.
Same issue with 1.5.4 on Centos 7.2 as well.

because the our log might be a little bit misleading, could you confirm whether you see redundant mounts for a pod volume such as secrete when you run "mount" command on the host? Thanks!
I'm seeing this with v1.5.2 running on AWS, setup via kops: The log has entries from the past ~4 days:


Question: Containers: []api.Container{ { Name:    "scrub-checker", <del> Image:   "busybox", </del> <ins> Image:   "gcr.io/google_containers/busybox", </ins> Command: []string{"/bin/sh"}, Args:    []string{"-c", "test ! -e /mnt/index.html || exit 1"}, VolumeMounts: []api.VolumeMount{

Testgrid; https://k8s- Sample Failure: The problem seems to be that Kube-anywhere isn't working since commit /sig cluster-lifecycle /kind failing-test /priority important-soon cc:
/milestone v1.13
sent possible fix PR:
sent new PR might get merged once github is back on track.
Reopening, because the test isn't passing yet. /reopen
Reopening this issue. <detailsIn response to : Reopening, because the test isn't passing yet. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
true, i was almost sure that the solution in that last PR won't be sufficient... will investigate more tomorrow.
/assign
I see https://k8s- job failing as well. Will the 2 open PRs address this failure as well?
all the kubeadm-gce-x tests are broken at the moment due to the e2e refactor. we will hopefully decide on how to approach the issue today.
latest efforts:
the tests are working on the cluster up/down side, except that they fail when dumping the node logs. looking into it.
should be green by tomorrow.
Looks good, thanks! One flake in the last run - let's keep an eye on it but it should be fine. /cc /close
Closing this issue. <detailsIn response to : Looks good, thanks! One flake in the last run - let's keep an eye on it but it should be fine. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "os" "path/filepath" "strings" <ins> "sync" </ins> "github.com/golang/glog" )

As per We should look for the .dockercfg in three places: ./.dockercfg, ~/.dockercfg, and $rootDir/.dockercfg
I'll tackle this.
Bunch of questions: we also add a flag that overrides the search mechanism suggested in this issue? Perhaps, ? to the user's homedir requires we use os/user [0] . The os/user package and cross-compiling don't play nice [1]. So, if we add to the search path, cross-compiled kubelet binaries will not actually be able to take advantage of it. Is this an issue? is the proper lookup order? I suggest: if provided, , , . walking through the lookups, do we stop at the first file that exists, or the first file that exists and is parsed as a valid ? [0] [1] /cc
I would absolutely love a --dockercfg option, and it would also be helpful if it logged which .dockercfg it ultimately used.
1) Seems like YAGNI to me 2) Is all of os/user broken? We will eventually want actual user support. I'm fine with dropping it for now. 3) I would say $rootDir/.dockercfg, ./.dockercfg (assuming we drop ~) 4) First successful parse sounds good to me. And can we NOT log errors if files don't exist :)
would it make sense to default to the kubelet config directory if provided rather than the working directory of the kubelet process? then no extra flag required.
What ever happened with this idea?
/cc
It never got high enough on my weekend TODO list :( I can keep telling myself that I need to do it, or we can find someone who has a bit more time than me.
I can probably get to this today.
Cool, thank you! , Mike Danese wrote:


Question: kube::test::get_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' kubectl delete pvc b-pvc 2>&1 "${kube_flags[@]}" <del> ## kubectl apply --prune --prune-whitelist(-w) </del> <ins> ## kubectl apply --prune --prune-whitelist </ins> # Pre-Condition: no POD exists kube::test::get_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' # apply pod a

When writing the docs for , I found there is a flag with short hand flag has been add to in 1.5. The flag is mostly used by add-on manger. I think we only give the flag that is very commonly used a shorthand flag. I think we should keep and save it for other flags in the future. We can make the change before it goes out in 1.5. cc:


Question: "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/util/mount" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util/node" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/volume" "github.com/golang/glog" )

etcd cluster backing apiserver died spontaneously, after restart apiserver pulled up the pods on different machines. Node storage was backed by ext4 over rbd, which became mounted on several nodes at once. That could have possibly led to data corruption. I guess that the proper way to mount rbd volume is to take a lock on it first, which will at least fail if it's locked somewhere else, requiring manual intervention, but not destroying data. The lock should be optional, in case a proper cluster fs is mounted over the rbd. In this case there was clearly no need for accessing the data from several pods so ext4 is deemed a proper fs.
, Vladimir Pouzanov < :
want to get some detail. How is your pod-to-rbd mapping like? Since rbd image is a block device, it can be used by one pod at a time. I am not sure if locking is a proper way to go - if the pod holds a lock but dies in the middle of mounting, the rbd image (or any other volume type) is then excluded from access. In such case, a high level coordination at apiserver is a better place to resolve this problem.
Here's my rc: I guess it's better to have a lock to be cleaned up manually rather than to have non-cluster fs to be mounted in several locations at once?
thanks, this pod looks good to me. what would cause multiple nodes run the same pod?
I don't know the exact reason of why kubelet didn't unmount the old fs, but after apiserver and scheduler were restarted pod got allocated on different node.
It is a scary thing to see etcd down, it is meant to be reliable...
Unfortunately I don't have long-term logging in test cluster so I have no idea why it actually died. but using kubedns as primary node resolver was clearly a bad idea, once it died, nodes stuck with no resolver at all.
your thought on HA?
Wrong Tim St. Clair?
- you HAVE to be ready to handle network partitions - the old node might still be alive and even using the volume, but the apiserver can't see the node, so it thinks it deletes the pod and starts it elsewhere. If RBD corrupts itslef in this case, it is not useful and we should strip it from the codebase.

, Huamin Chen wrote:
that makes sense. rbd doesn't corrupt itself. This is inconsistency happens when you have two pods mounting the same rbd image. How does GCE PD handle/prevent this case?
It doesn't allow more than one read-write attachment. Multiple read-only attachments is OK. Better to fail than to allow corruption. , Huamin Chen wrote:
That's neat. I have a lock fencing for rbd, testing now.


Question: m.thresholdsFirstObservedAt = thresholdsFirstObservedAt m.nodeConditionsLastObservedAt = nodeConditionsLastObservedAt m.thresholdsMet = thresholds <ins> // determine the set of thresholds whose stats have been updated since the last sync thresholds = thresholdsUpdatedStats(thresholds, observations, m.lastObservations) debugLogThresholdsWithObservation("thresholds - updated stats", thresholds, observations) </ins> m.lastObservations = observations m.Unlock()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: unused images does not get deleted What you expected to happen: unused images get deleted How to reproduce it (as minimally and precisely as possible): configure kubelet with these args: Anything else we need to know?: when using above config, the node's events history showing: If I change to it works, node's events history showing these: The difference is there is an extra event, which is not seen with cofnig. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): Linux 192-168-1-1 4.4.0-63-generic -Ubuntu SMP Wed Feb 1 17:20:32 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig area/kubelet
Maybe we are with the same problem. Kubelet Parameters for tests: When we reach a disk consumption of 20% (for debug) - after 2 minutes, the deletion of logs, image docker, and then PODS should begin. But nothing happens. The debug logs kubelet saw the following: Problems are observed in the versions 1.6.X, 1.7.X In version 1.5.7, it looks like this: Can you help us? Or we help you, with Debug? Environment: Kubernetes version (use kubectl version): OS (e.g. from /etc/os-release): ubuntu 16.04.2 LTS Kernel (e.g. uname -a): Linux node2 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x8664 GNU/Linux Docker: Docker version 1.11.2, build b9f10c9_ PS We tried the rollback option from , but it did not change anything.
hello! Can you spare a little time to this problem?
Well, here is what we have found so far. In our setup, eviction.managerImpl.synchronize() frequently runs with a set of signalObservation objects, which contains the same data as in previous runs, even for "time" field. Probably this is due to slow cadvisor. Unfortunately, eviction.thresholdsUpdatedStats() considers such data expired and removes it from the list of observed thresholds. This was always the case, but before commit "time" field wasn't initialized, and thresholdsUpdatedStats allowed it to persists between iterations of synchronize() by explicit check for After neither check for zero, no check for one time being more recent then another, doesn't work as iterations are frequently performed with the same summary.Node.Fs object. And here we are with blinking "node conditions - observed: [DiskPressure]" in logs: one run of synchronize sets it, and next one discards it (runs are delimited with "synchronize housekeeping" message): The question is whether this logic in thresholdsUpdatedStats is correct now or not. Should garbage collection work even with slow cadvisor (if this is the reason of slow updates of summary.Node.Fs)?

Probably frequency of synchronize should be adjusted, or summary.Node.Fs updates should be synchronized with execution of synchronize()? We can correct the problem by monkey-patching makeSignalObservations and get rid of summary.Node.Fs.Time usage in signalObservation, but is it right?
The 2 minute grace period has not been met, so no action is taken. This is the difference between a "soft" and "hard" threshold: The soft only takes action after the threshold has been met for the specified period of time. If no new stats exist, the correct behavior is not to take action. This could delay action by 10 seconds, but it is better not to be overactive when evicting (we have had this problem in the past). The eviction manager assumes that disk pressure still exists when it does not have new stats. It basically just "skips" that synchronize call when that is the case.
dekhtyarev@ it looks like your node goes in and out of disk pressure. I see messages woven into your logs.
Hmmm, or maybe there is something here. Looks like the timer is reset for soft eviction thresholds.
No, On seeing this, I just ran several PODs on the node and did to achieve the conditions of eviction. I used ~22% of the disk, at the threshold of 80%. And disk consumption on the node did not change.
We just think that the timer resets, as the timestamp of the received data has not changed and this is taken into account in the eviction manager logic. Look at the field here: wrote about it, I think.
I see the issue. Ill try and get a bugfix into 1.8.
cc Soft Eviction thresholds have not been functional since 1.6...
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: return os.bsOpts.TrustDevicePath } <ins> // GetLabelsForVolume implements PVLabeler.GetLabelsForVolume func (os *OpenStack) GetLabelsForVolume(pv *v1.PersistentVolume) (map[string]string, error) { // Ignore any volumes that are being provisioned if pv.Spec.Cinder.VolumeID == k8s_volume.ProvisionedVolumeName { return nil, nil } // Get Volume volume, err := os.getVolume(pv.Spec.Cinder.VolumeID) if err != nil { return nil, err } // Contruct Volume Labels labels := make(map[string]string) labels[kubeletapis.LabelZoneFailureDomain] = volume.AvailabilityZone labels[kubeletapis.LabelZoneRegion] = os.region glog.V(4).Infof("The Volume %s has labels %v", pv.Spec.Cinder.VolumeID, labels) return labels, nil } </ins> // recordOpenstackOperationMetric records openstack operation metrics func recordOpenstackOperationMetric(operation string, timeTaken float64, err error) { if err != nil {

Resizing a persistent volume is partially broken on openstack (cinder) with k8s v1.11.1. Partially in the sense that static resizing (requiring a pod restart) works, but the newly introduced online resizing (not requiring a pod restart) fails. Upon resizing a pvc it remains in the Resizing condition forever (see the warning event) until I delete the pod that has this pvc mounted: After pod deletion it notifies me (correctly) that it's now waiting for the pod to come up to finish resizing: So far the warning event does not break the ExpandPersistentVolumes feature, as this one anyhow requires a pod restart. However it breaks the newly introduced ExpandInUsePersistentVolumes feature (alpha with k8s v1.11; doesn't require a pod restart for resizing to finish), as the pvc will remain in "Resizing" condition forever (or until the pod is restarted but this defeats the purpose of ExpandInUsePersistentVolumes). I'm not sure whether this is fully related but upon digging around I found and a consecutive fix with which was merged on Jan 23. However a few days later this fix seems to be (probably accidentally) overwritten with this commit: /kind bug /sig storage /sig openstack
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
It is already fixed. So I will close it.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am trying cinder online pvc resizing on kubernetes1.12.2. I noticed this kubernetes version has include your fix, but I still failed to resize pvc online.

The error log has no change: Warning VolumeResizeFailed 45s (x2 over 66s) volumeexpand Error expanding volume "default/myclaim-expend" of plugin : volume status is not available The error log is reported below code line "volume.Status != volumeAvailableStatus" // ExpandVolume expands the size of specific cinder volume (in GiB) func (os *OpenStack) ExpandVolume(volumeID string, oldSize resource.Quantity, newSize resource.Quantity) (resource.Quantity, error) { volume, err := os.getVolume(volumeID) if err != nil { return oldSize, err } if volume.Status != volumeAvailableStatus { // cinder volume can not be expanded if its status is not available return oldSize, fmt.Errorf("volume status is not available") } One point confused me, once cinder volume is used by pod, it's status is "in-use", how the online pvc resizing pass the code check "volume.Status != volumeAvailableStatus"? There's where the online pvc resizing fails.
It looks like Cinder can not support online volume expansion - which is what is happening here. Even if online expansion is enabled, I think to actually perform resizing of Cinder volumes - you must bring the pod down.


Question: cleanupServerPodAndService(f, podServer, service) }) <del> ginkgo.It("should support a 'default-deny' policy [Feature:NetworkPolicy]", func() { </del> <ins> ginkgo.It("should support a 'default-deny-ingress' policy [Feature:NetworkPolicy]", func() { </ins> policy := &networkingv1.NetworkPolicy{ ObjectMeta: metav1.ObjectMeta{ <del> Name: "deny-all", </del> <ins> Name: "deny-ingress", </ins> }, Spec: networkingv1.NetworkPolicySpec{ PodSelector: metav1.LabelSelector{},

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We are missing a test which specifies Ingress,Egress (both) for network policys (in ). Why is this needed: So that our network policy test coverage matches the semantics of what is commented and supported in the API.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
/assign
/assign I'd like to do this if you don't mind
Sure go ahead. I hadn't gotten to it for a while. Appreciate you picking this up


Question: verbose      = flag.Bool("verbose", false, "If true, print extra information") proxy        = flag.Bool("proxy", false, "If true, run a proxy to the api server") www          = flag.String("www", "", "If -proxy is true, use this directory to serve static files") <ins> templateFile = flag.String("template_file", "", "If present load this file as a golang template and us it for output printing") templateStr  = flag.String("template", "", "If present parse this string as a golang template and us it for output printing") </ins> ) func usage() {

To remove dependence on jq and generally make scripting much easier, we should make another output flag which takes a formatter. We can use go's template package, which is very flexible.


Question: // the labelSelector in the specified namespaces, where co-located is defined as running on a node // whose value of the label with key topologyKey matches that of any node on which any of the // selected pods is running. <del> // For PreferredDuringScheduling pod anti-affinity, empty topologyKey is interpreted as "all topologies" // ("all topologies" here means all the topologyKeys indicated by scheduler command-line argument --failure-domains); // for affinity and for RequiredDuringScheduling pod anti-affinity, empty topologyKey is not allowed. // +optional </del> <ins> // Empty topologyKey is not allowed. </ins> optional string topologyKey = 3; }

The field is marked as optional: The comment also describes a use case for empty topologykey: But the validation rule says it's required:
cc
cc could you take a look? Thank you. The validation rule is by
the release note of should go into the "action-required" section of the , but it's currently in "Other notable changes".
/assign
It seems that we should also update the external api type comments. Right?
I will look into it today.
If you didn't work on this, I am pleasure to send PR to fix this. :)
yeah sure. thanks.


Question: for predKey, result := range predicateResults { // update equivalence cache with newly computed fit & reasons // TODO(resouer) should we do this in another thread? any race? <del> ecache.UpdateCachedPredicateItem(pod.GetName(), nodeName, predKey, result.Fit, result.FailReasons, equivalenceHash) </del> <ins> ecache.UpdateCachedPredicateItem(pod.GetName(), nodeName, predKey, result.Fit, result.FailReasons, equivCacheInfo.hash) </ins> } } return len(failedPredicates) == 0, failedPredicates, nil

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Today, . This seems unnecessary and should be done once for all the nodes. Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig scheduling cc:
/assign Thanks, will send fix per slack discussion.
/cc


Question: MUST now include the new Label. release: v1.18 file: test/e2e/apimachinery/namespace.go <ins> - testname: ResourceQuota, apply changes to a ResourceQuota status codename: '[sig-api-machinery] ResourceQuota should apply changes to a resourcequota status [Conformance]' description: Attempt to create a ResourceQuota for CPU and Memory quota limits. Creation MUST be successful. Updating the hard status values MUST succeed and the new values MUST be found. The reported hard status values MUST equal the spec hard values. Patching the spec hard values MUST succeed and the new values MUST be found. Patching the hard status values MUST succeed. The reported hard status values MUST equal the new spec hard values. Getting the /status MUST succeed and the reported hard status values MUST equal the spec hard values. Repatching the hard status values MUST succeed. The spec spec MUST NOT be changed when patching /status. release: v1.26 file: test/e2e/apimachinery/resource_quota.go </ins> - testname: ResourceQuota, update and delete codename: '[sig-api-machinery] ResourceQuota should be able to update and delete ResourceQuota. [Conformance]'

[X] Test approval issue: [X] Test PR: 19 Oct 2022 [x] Two weeks soak end date: 2 Nov 2022 [x] Test promotion PR: According to this APIsnoop query, there are still three remaining ResourceQuota status endpoints which are untested. patchCoreV1NamespacedResourceQuotaStatusreplaceCoreV1NamespacedResourceQuotaStatusreadCoreV1NamespacedResourceQuotaStatus Using a number of existing e2e test practices a new has been created for 3 ResourceQuota Status endpoints. The e2e logs for this test are listed below. This query shows which Resource Status endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 3 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func isRegistryEqual(registryX, registryY Registry) bool { for name, pluginFactory := range registryY { if val, ok := registryX[name]; ok { <del> if reflect.ValueOf(pluginFactory) != reflect.ValueOf(val) { </del> <ins> if reflect.ValueOf(pluginFactory).Pointer() != reflect.ValueOf(val).Pointer() { </ins> // pluginFactory functions are not the same. return false }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Below are the failures while running unit test with golang in master branch. Reference : $ go version go version devel go1.17- Thu Apr 22 01:40:02 2021 +0000 linux/ppc64le Note: Occurs in x86 architecture as well. The tests should Pass as with stable go git clone cd kubernetes make test KUBERACE=-race WHAT=./pkg/scheduler/framework/runtime/ Master branch of kubernetes repo. OS: Linux Observed failure on ppc64le and x86 architectures Below is the attached log:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I request help on getting this triaged. Is there a procedure to follow to get this accepted as a bug?
The Go compiler developer has identified the bugs in the test that is causing it to fail. They are listed in the last comment in the Go issue before closing it.
So the correct way to compare function would be something like would you test it in your env?
The issue still exists. Can you please reopen?
I am not sure if you can access this link for our CI job. Can you please check? https://prow.ppc64le- The above tests have been failing every day in our jobs!
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
downloads go from url="${GOLANGVERSION}.linux-" downloads go from "curl -X GET http://build-?project=build_commit -o " I suspect the postsubmit-master-golang-kubernetes-unit-test-ppc64le failed for wrong golang version.

failed for other problems. The TestMerge is not failed. For ppc64le failures, there are several issues in golang: (fixed yesterday)
Yes the go versions for both the jobs are different and it is intentional: : It is meant to run Kubernetes unit tests with the stable version of golang : It is meant to run Kubernetes unit tests with golang built from the master branch of As I mentioned in the description of this issue, these unit test cases are failing only on golang that is built from master branch and passing with stable versions of golang. And as Lynn mentioned in , the golang team has concluded that these failures are due to the bugs in the test case. And the golang issues that you mentioned do not seem to be related to the failures mentioned in this issue.
With , is false Per , do you mean that we should change to ?
No. In general comparing is the right thing to do. But if the Value is actually a , which is not comparable, comparing will panic.
/sig scheduling
Note from golang team about this bug is here:


Question: base.Add(base, big.NewInt(1)) max-- <ins> // cidr with whole mask can be negative if max < 0 { max = 0 } </ins> r := Range{ net:     cidr, base:    base,

kube-apiserver observed a panic: kube-apiserver should not panic Not sure yet -- will add more details as I investigate. No response v1.25.4 N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /cc
max is greater than offset?
/assign


Question: import ( "context" "fmt" <del> "os" </del> <ins> "strconv" </ins> "strings" v1 "k8s.io/api/core/v1"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider Instances interfaces don't work expected on Windows containers. This is because Windows containers don't support HostNetwork mode, while Azure cloud provider depends on "os.Hostname()" to check whether the VMName from IMDS is same with NodeName: To fix it, the dependency on Hostname should be removed. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /priority critical-urgent /area provider/azure /milestone v1.18