Question: // by kubelet-flags. "--feature-gates", framework.TestContext.FeatureGates, "--eviction-hard", "memory.available<250Mi,nodefs.available<10%,nodefs.inodesFree<5%", // The hard eviction thresholds. <ins> "--eviction-minimum-reclaim", "nodefs.available=5%,nodefs.inodesFree=5%", // The minimum reclaimed resources after eviction. </ins> "--v", LOG_VERBOSITY_LEVEL, "--logtostderr", ) // Enable kubenet by default.

Clusters on GCE set eviction thresholds for disk and inodes. Given that reclaiming disk and inodes is slow, we should set minimum reclaim thresholds too. Essentially, set . I assume percentages are supported for min reclaims too. Do we have system tests for min reclaim? cc
I just took a glance at the parser for eviction minimum reclaim, it doesn't look like it tries to parse percentages. can you confirm?
did percentage support, and we did not extend it to min reclaim. It's a good idea to do though. On Tuesday, September 13, 2016, Michael Taufen wrote:
Hmm.. In that case for v1.4, can we set the min reclaim to ? What values are you planning on using in Open Shift ?
This should fix this issue to avoid node condition being flip-flop, but I think it shouldn't block 1.4 release.
More than flip-flop, it will help the node recover from disk pressure condition sooner. , Dawn Chen wrote:
/cc
Spawned to add percentage support for min reclaim. I will follow up with our proposed default for nodefs later
This needs to be triaged as a release-blocker or not for 1.5
can you check if we have set default values for disk and inode based evictions in kubelet? If not, we should add them.
I believe our current default value for the --eviction-hard flag is "memory.available<100Mi", found in Our tests currently run using: "memory.available<250Mi,nodefs.available<10%,nodefs.inodesFree<5%", found in What should I change the default to? This issue is about minimum reclaim, should I look into changing those flags as well?
It doesn't appear that we have defaults for minimum reclaim, should I add minimum reclaim for memory in addition to disk and inodes?
I think GCE defaults do not apply globally, we should change what GCE does, and handle setting global defaults in 1.6.
the global defaults are expected to be based on our OSS tests. Our original plan was to roll out disk eviction by default in v1.5. This is not specific to GCE.
Can you make a call on if this is a blocker or non-blocker for 1.5 by EOD Friday please


Question: if kubeClient != nil { // TODO: cache.NewListWatchFromClient is limited as it takes a client implementation rather // than an interface. There is no way to construct a list+watcher using resource name. <del> fieldSelector := fields.Set{"metadata.name": hostname}.AsSelector() </del> <ins> fieldSelector := fields.Set{client.ObjectNameField: hostname}.AsSelector() </ins> listWatch := &cache.ListWatch{ ListFunc: func() (runtime.Object, error) { // TODO: Use List() with fieldSelector when it is supported.

My broke it for v1beta3, then fixed it for v1beta3 and broke it for v1beta1. I'm working on a fix.
Honestly I'm very surprised that more people aren't complaining-- integration test logs are full of noise because of this.
Exact text of error log is "Failed to watch *api.Node: field label not supported: "
cc


Question: }, }, }, <del> Required: []string{"replicas", "availableReplicas"}, </del> <ins> Required: []string{"replicas"}, </ins> }, }, Dependencies: []string{

Looking at the for , it defines as a required attribute. But looking at the of , it states . Should this in fact be a required attribute? I come about this from using , a python package that generates models from the swagger definitions and thus builds a that requires an attribute that is not enabled. That only truly required attributes be required in the swagger definition Reproduction is not applicable as this is an issue with a spec No response Appeared in 1.22 (I believe) and affects current versions. Does not affect 1.21 n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
alpha: has that is planned for removal in beta. ( results in fields as non-required in the OpenAPI) beta: removes the this might be a question whether feature gate controlled fields must remain until the feature gate is removed (and how would validation work in that case). ~old feature gates graduations to beta enabled a feature by default.~ a topic related to ? but it does not apply to feature gates. /kind documentation /sig apps
cc (since you are the author)
new fields to existing types must always be marked as +optional, since clients are not required to specify them
please open a PR adding back the +optional tag, regenerate openapi, and open picks back to 1.23 and 1.22


Question: } <ins> type fakeQueue struct { workqueue.RateLimitingInterface t   time.Duration key interface{} } func (f *fakeQueue) AddAfter(key interface{}, t time.Duration) { f.t = t f.key = key } </ins> // this test will take around 61 seconds to complete func TestController2_updateCronJob(t *testing.T) { cjc := &fakeCJControl{} jc := &fakeJobControl{} type fields struct { <del> queue          workqueue.RateLimitingInterface </del> <ins> queue          *fakeQueue </ins> recorder       record.EventRecorder jobControl     jobControlInterface cronJobControl cjControlInterface

pull-kubernetes-unit TestController2updateCronJob/spec.schedulechanged TestController2_updateCronJob /sig apps /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is important to resolve before promotion to v1
looks like the cause was
This function should be related.
Another one:
When the flake happen, there are messages like When the system is overloaded, it would happen. is trying to replace the current queue in tests with a mock.
Yeah, I spoke with to make sure we use a mock of the queue so we only get the most important bit of information ignoring the remaining parts


Question: if err := addPriorityClasses(ctrl, test.classesAfter); err != nil { t.Errorf("Test %q: unable to add object to informer: %v", test.name, err) } <del> defaultPriority, err = ctrl.getDefaultPriority() </del> <ins> pcName, defaultPriority, err = ctrl.getDefaultPriority() </ins> if err != nil { t.Errorf("Test %q: unexpected error while getting default priority: %v", test.name, err) } <del> if err == nil && defaultPriority != test.expectedDefaultAfter { t.Errorf("Test %q: expected default priority %d, but got %d", test.name, test.expectedDefaultAfter, defaultPriority) </del> <ins> if err == nil && (defaultPriority != test.expectedDefaultAfter || pcName != test.expectedDefaultNameAfter) { t.Errorf("Test %q: expected default priority %s(%d), but got %s(%d)", test.name, test.expectedDefaultNameAfter, test.expectedDefaultAfter, pcName, defaultPriority) </ins> } } }

Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup /sig scheduling Description: If Pod did not set , admission controller will set to "default value". If the default value are from default priority class, it's better to also set of pod.
can I take this? /assign
, thanks :) I created a PR for functionality, but that'll be great if you can help add some e2e test for it. I can help to shepherd :)
thanks let me have a try :)


Question: errCh := decorated2.Do(func() { defer close(executed) if !delegated { <del> t.Fatalf("expected delegated execution") </del> <ins> t.Errorf("expected delegated execution") </ins> } errOnce.Report(expectedErr) })

Can we get this fixed ASAP? Thanks!
yep, thanks for reporting , Brendan Burns wrote:
how reproduceable is this? I have some changes that I think will resolve it but I couldn't reproduce locally even when all my cpu's were pretty tied up and using a high GOMAXPROCS.
flakey label process?
the labelling is due to me failing @ github's label interface. Bulk editing labels removes them if any of the selected things have the label sigh Thanks for the quick fix.


Question: } } <ins> // TestMultipleServiceChanges tests that endpoints that are not created because of an out of sync endpoints cache are eventually recreated // A service will be created. After the endpoints exist, the service will be deleted and the endpoints will not be deleted from the cache immediately. // After the service is recreated, the endpoints will be deleted replicating an out of sync cache. Expect that eventually the endpoints will be recreated. func TestMultipleServiceChanges(t *testing.T) { ns := metav1.NamespaceDefault expectedSubsets := []v1.EndpointSubset{{ Addresses: []v1.EndpointAddress{ {IP: "1.2.3.4", NodeName: &emptyNodeName, TargetRef: &v1.ObjectReference{Kind: "Pod", Name: "pod0", Namespace: ns}}, }, }} endpoint := &v1.Endpoints{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: ns, ResourceVersion: "1"}, Subsets:    expectedSubsets, } controller := &endpointController{} blockDelete := make(chan struct{}) blockNextAction := make(chan struct{}) stopChan := make(chan struct{}) testServer := makeBlockingEndpointDeleteTestServer(t, controller, endpoint, blockDelete, blockNextAction, ns) defer testServer.Close() *controller = *newController(testServer.URL, 0*time.Second) addPods(controller.podStore, ns, 1, 1, 0, ipv4only) go func() { controller.Run(1, stopChan) }() svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: ns}, Spec: v1.ServiceSpec{ Selector:  map[string]string{"foo": "bar"}, ClusterIP: "None", Ports:     nil, }, } controller.serviceStore.Add(svc) controller.onServiceUpdate(svc) // blockNextAction should eventually unblock once server gets endpoint request. waitForChanReceive(t, 1*time.Second, blockNextAction, "Service Add should have caused a request to be sent to the test server") controller.serviceStore.Delete(svc) controller.onServiceDelete(svc) waitForChanReceive(t, 1*time.Second, blockNextAction, "Service Delete should have caused a request to be sent to the test server") // If endpoints cache has not updated before service update is registered // Services add will not trigger a Create endpoint request. controller.serviceStore.Add(svc) controller.onServiceUpdate(svc) // Ensure the work queue has been processed by looping for up to a second to prevent flakes. wait.PollImmediate(50*time.Millisecond, 1*time.Second, func() (bool, error) { return controller.queue.Len() == 0, nil }) // Cause test server to delete endpoints close(blockDelete) waitForChanReceive(t, 1*time.Second, blockNextAction, "Endpoint should have been recreated") close(blockNextAction) close(stopChan) } func TestEndpointsDeletionEvents(t *testing.T) { ns := metav1.NamespaceDefault testServer, _ := makeTestServer(t, ns) defer testServer.Close() controller := newController(testServer.URL, 0) store := controller.endpointsStore ep1 := &v1.Endpoints{ ObjectMeta: metav1.ObjectMeta{ Name:            "foo", Namespace:       ns, ResourceVersion: "rv1", }, } // Test Unexpected and Expected Deletes store.Delete(ep1) controller.onEndpointsDelete(ep1) if controller.queue.Len() != 1 { t.Errorf("Expected one service to be in the queue, found %d", controller.queue.Len()) } } </ins> func stringVal(str *string) string { if str == nil { return "nil" } return *str } <ins> // waitForChanReceive blocks up to the timeout waiting for the receivingChan to receive func waitForChanReceive(t *testing.T, timeout time.Duration, receivingChan chan struct{}, errorMsg string) { timer := time.NewTimer(timeout) select { case <-timer.C: t.Errorf(errorMsg) case <-receivingChan: } } </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? no. What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): service endpoints missing Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.6.2 Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Install tools: manual Others: What happened: Service in namespace sees no endpoints that are available What you expected to happen: For it to see available endpoints How to reproduce it (as minimally and precisely as possible): Not sure I can. It appeared on the latest remove/replace of worker nodes, but was fine before Anything else we need to know: I have a traefik Ingress controller defined and a service for it: Selectors match, the pods are running just fine, yet the endpoints do not show up. I suspect if I delete the service and recreate it, it probably will pick it up, but I shouldn't have to, and am concerned it might occur again; rather actually debug it. How can I check what kube is doing to affiliate it? How often does it check? Where?
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
The system is asking for a SIG, but it isn't clear which SIG is appropriate for this. Taking best guess, apologies if I am wrong.

/sig sig/apps /sig sig/scheduling
/sig network
can you provide the following output?
I wish I could... in the week since, the cluster has been deleted and recreated several times. AWS instances may be cheap, but still aren't free. Client didn't want them sticking around too long for the cost.
Should we close this issue pending more information or keep it open for someone to test out?
we are killing and recreating clusters with some regularity over these weeks. I am happy if you want to close it since I cannot recreate it yet. Just let me reopen it if I manage to recreate and get more info. Just tell me, please, if it does come up again, what additional data should I get? There isn't much data in the docs on how the service resolves what endpoints it should use and how to figure out why it is misbehaving when it does. Happy to take pointers.
Please reopen when it happens again. Please record related endpoints objects and pod objects.
I'm facing same issue with Nginx by trying to expose the nodePort for Nginx-ingress-controller pod
Try .
Your service is using while your pods are using .
Sorry, didn't notice that
OS: ubuntu 16.04 kubernetes version: v1.8.4 hi, I have the same problem now. $ kc get endpoints -njenkins This situation has been going on for a long time and no solution has been found. Help out. Thanks
Same the problem, svc lost the endpoint after a few time. If deleting the pod, then the svc endpoint is show up again. How to debug this problem?
Im having the same problem on Windows Nodes, after some hours all svc on one node loses their endpoints. Kubernetes V1.15
May be fixed by this?

same issue for a custom helm that I created: kubectl version Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-19T10:52:43Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-04-03T04:12:37Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} kubectl describe service/vnfdb-ui Name: vnfdb-ui Namespace: test-helm Labels: Annotations: <noneSelector: , Type: LoadBalancer IP: 10.105.168.158 Port: http 80/TCP TargetPort: http/TCP NodePort: http /TCP Endpoints: <noneSession Affinity: None External Traffic Policy: Cluster Events: <none$ kubectl get pods -l , NAME READY STATUS RESTARTS AGE vnfdb-ui--ttrwm 1/1 Running 0 9m $ kubectl get endpoints vnfdb-ui NAME ENDPOINTS AGE vnfdb-ui <none9m33s $ kubectl get endpoints vnfdb-ui -o yaml apiVersion: v1 kind: Endpoints metadata: creationTimestamp: "2019-08-24T21:28:58Z" labels: vnfdb Tiller ui "1.0" ui-0.1.0 name: vnfdb-ui namespace: test-helm resourceVersion: "" selfLink: /api/v1/namespaces/test-helm/endpoints/vnfdb-ui uid: -c6b6-11e9-a0d5- Any clue what is wrong here and how can I fix it?
Got it, apparently realized that its not only the selector that has to match labels. The port names should be the same in the pod definition and services definition as well. The latter was the culprit in my case.
Same the problem.My service and pod is running. But the service do not has endpoint,and this situation lasts for 10 hours. It is OK after I restart controller manager.

My last operation was to delete the service and create the service again. I have not done any other operations since then. Kubernetes V1.11.5
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Good catch!
Happens to us once in a while: We have a process creating a service, then a test deleting that same service, which ends up being automatically recreated. Once in a while (every ~100 test), the re-created service never gets any endpoints, even after waiting for several hours. I'm fairly confident recreating the service again fixes it, but that's not a solution. It just happened on a GKE cluster in version , for which I don't have controller-manager logs available unfortunately. I'll try to reach GCP support, they may be able to troubleshoot more from the controller-manager logs. Edit: the endpoint magically appeared after 22 hours.
Thanks for the detailed bug report! I'll try to recreate this scenario with some new tests. /assign
A quick follow up to let you know we haven't forgotten about this issue. From what we can tell, this is being caused by an Endpoints informer cache that can potentially be out of date when syncService() runs. Essentially syncService() would be called twice - once for the delete and once for the create event.

There's a chance that when the second run of syncService() happens the Endpoints cache may not have been updated with the latest run. If that happens, syncService() will not make any changes because it sees an endpoints resource that has all the right ports and addresses set. is working on a fix here that will mirror what we've done for the EndpointSlice controller with the . This will involve watching Endpoints changes and calling syncService() if the change has not already been accounted for by the controller. As far as I can tell, this should be a relatively rare bug. It should only happen if the cache is out of date, the recreated Service is identical, and the Pods have not changed at all. Given the relative complexity of the fix here, I'm not sure how far we'll be able to backport this. At the very least, we're hoping to have a fix in for 1.19.
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think is almost ready, sorry for not catching the lifecycle close earlier.
Dude, you saved my life.


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" <del> "k8s.io/kubernetes/pkg/apis/apps" </del> <ins> extensionsinternal "k8s.io/kubernetes/pkg/apis/extensions" </ins> "k8s.io/kubernetes/test/e2e/framework" imageutils "k8s.io/kubernetes/test/utils/image"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug We can see all related with test failed. and so on. I think it maybe relate with PR and . What happened: Always fails. What you expected to happen: Pass all.
/cc PTAL. Thanks.
Why did the not pass the test but was still merged?
/priority important-soon /kind e2e-test-failure
/area platform/gce
Looks like the fix is in
is not a blocking / required test (for merge) currently IIRC, kubernetes/kubernetes uses the "submit-queue" which will only pay attention to some tests.
i think this is true, as confirmed by the current state of this PR: it fails the test, but the CI says instead of is there are way to see which tests would cause a ?
off the top of my head the definitive way is this list: but note that is working on moving us to in the nearish future, and every other kubernetes repo besides this one is either not using merge automation or is using , which behaves differently and has seperate config integrated with the rest of test-infra.
thanks for the details! :+1:


Question: function usage() { echo "!!! EXPERIMENTAL !!!" echo "" <del> echo "${0} [-M|-N|-P] -l | <version number or publication>" </del> <ins> echo "${0} [-M|-N|-P] -l -o | <version number or publication>" </ins> echo "  Upgrades master and nodes by default" echo "  -M:  Upgrade master only" echo "  -N:  Upgrade nodes only" echo "  -P:  Node upgrade prerequisites only (create a new instance template)" <ins> echo "  -o:  Use os distro sepcified in KUBE_NODE_OS_DISTRIBUTION for new nodes" </ins> echo "  -l:  Use local(dev) binaries" echo "" echo '  Version number or publication is either a proper version number'

During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm.

If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.


Question: "containers": [ { "name": "cluster-autoscaler", <del> "image": "gcr.io/google_containers/cluster-autoscaler:v0.5.0", </del> <ins> "image": "gcr.io/google_containers/cluster-autoscaler:v0.5.1", </ins> "command": [ "./run.sh", "--kubernetes=http://127.0.0.1:8080?inClusterConfig=f",

On some occasions Cluster Autoscaler may scale up the cluster two times. This results in growing the cluster too large. The size gets back to normal soon but the user will be forced to pay for the unneeded node for 10min/1h, depending on the cloud provider. cc:
Please provide more information.
Imagine that you have just created a pod that cannot be scheduled. CA expands one of the migs to accommodate it. Just when the node arrives there is a brief moment when the pod is still pending (scheduler didn't retry it yet) but the node is available. Scale up is over. Pod is pending. So we might try to scale up again, which would be incorrect. To protect against such situations we run a simulated scheduling on all nodes. If any node can accommodate any of the pending pods then the pod is not pending and should not trigger a scale up. After a scale up we create "virtual" nodes inside CA so that we don't scale up again for the same pods. Once the node arrives its virtual placeholder is deleted. And, if I'm correct, we have a bug here. A node with node condition ready=true but with non ready network is considered as fully started but broken. Its virtual placeholder is gone so we trigger scale up again. Seconds later the network arrives and the scale up appears to be non-needed. I'm testing right now a 10 line fix for this issue. After the fix is merged I would like to bump CA version in 1.6 as having the old code would impact all of the current CA users on GCE and GKE (and possibly on other cloud providers too).
is this a regression in cluster autoscaler for 1.6, or did the same behavior exist in 1.5?
It is a regression. Doesn't happen always (depends on the network setup timings) but when it does it is quite confusing.


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package container import ( "encoding/json" "testing" "k8s.io/api/core/v1" ) var ( sampleContainer = ` { "name": "test_container", "image": "foo/image:v1", "command": [ "/bin/testcmd" ], "args": [ "/bin/sh", "-c", "echo abc" ], "ports": [ { "containerPort": 8001 } ], "env": [ { "name": "ENV_FOO", "value": "bar" }, { "name": "ENV_BAR", "valueFrom": { "secretKeyRef": { "name": "foo", "key": "bar", "optional": true } } } ], "resources": { "limits": { "foo": "1G" }, "requests": { "foo": "500M" } } } ` sampleV115HashValue = uint64(0x311670a) sampleV116HashValue = sampleV115HashValue ) func TestConsistentHashContainer(t *testing.T) { container := &v1.Container{} if err := json.Unmarshal([]byte(sampleContainer), container); err != nil { t.Error(err) } currentHash := HashContainer(container) if currentHash != sampleV116HashValue { t.Errorf("mismatched hash value with v1.16") } if currentHash != sampleV115HashValue { t.Errorf("mismatched hash value with v1.15") } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: During the update from v1.7.1 to v1.8.0 all docker containers were restarted, because container spec hash changed. What you expected to happen: No restart of the docker containers, because they did not change at all. How to reproduce it (as minimally and precisely as possible): get yourself a k8s cluster in v1.7.x update (at least the kubelet) to v1.8.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/area kubelet
I ran into this earlier today. I was able to update the control plane & kube-proxy to 1.8.1 without any problems. Once I upgraded the kubelet though it caused container restarts.
/sig apps /sig api-machinery
the log message one receives is at: The function, that calculates the hashes is at: The hash function seems to not have changed. I suspect changes in the object itself, which is at: One can see: but:
But the new field does not initiated, so the restart seems unreasonable.
the result i tested was that the spew pkg also dumps the new add field(with nil value) to the hasher. is the new field. so, if the container use , then new MountPropagationMode will be included into the hash input, the hash result changed. other new fields are in the same situation.
Do you have any idea about preventing this?
We've hit this as well. Caused a cluster-wide restart of every container.
i also hit this issue as well, when I am upgrading cluster from v1.7.4 to v1.8.5. especially, the kubelet version from v1.7.4 to v1.8.5
Still seems to happen in 1.9.0
I can confirm this. v1.9.0 has introduced a new field attribute for , which is causing the hash value get changed.
It will exist when we use docker. I remember had described it.

There are details in this issues below:
thanks for digging out the old issue! Yes, any new field to the container spec would cause the hash to change. This is not a new issue but new fields are to the container spec rarely (compared with pod spec) so it doesn't happen so often. As discussed in , a new way to generate the hash (e.g., using only updatable fields) or other method for kubelet to examine and compare the state of the container against the spec is needed. Calling this (i.e., container restart) out in the 1.9 release note would be nice.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/close
Just wondering, why don't we calculate the container hash by since container image is the only mutable field? I know we have to do some refactoring to , but wouldn't it be more reliable?


Question: allErrs = append(allErrs, apivalidation.ValidateObjectMeta(&obj.ObjectMeta, false, ValidateThirdPartyResourceName, field.NewPath("metadata"))...) versions := sets.String{} <ins> if len(obj.Versions) == 0 { allErrs = append(allErrs, field.Required(field.NewPath("versions"), "must specify at least one version")) } </ins> for ix := range obj.Versions { version := &obj.Versions[ix] if len(version.Name) == 0 {

FYI. Currently the field of TPR is marked as which is wrong, actually if you try to create a TPR with empty versions, it will cause api server panic. Versions should be necessary field of TPR, we should validate this somehow.


Question: { node: api.Node{ ObjectMeta: metav1.ObjectMeta{ <ins> Name:   "foo10", Labels: map[string]string{"node-role.kubernetes.io/master": "", "kubernetes.io/role": "node", "kubeadm.alpha.kubernetes.io/role": "node"}, }, }, status: "Unknown,Master", }, { node: api.Node{ ObjectMeta: metav1.ObjectMeta{ Name:   "foo11", Labels: map[string]string{"kubernetes.io/role": "node", "kubeadm.alpha.kubernetes.io/role": "node"}, }, }, status: "Unknown,Node", }, { node: api.Node{ ObjectMeta: metav1.ObjectMeta{ </ins> Name:   "foo12", Labels: map[string]string{"kubeadm.alpha.kubernetes.io/role": "node"}, }, Status: api.NodeStatus{Conditions: []api.NodeCondition{{Type: api.NodeReady, Status: api.ConditionTrue}}}, }, <del> status: "Ready,node", </del> <ins> status: "Ready,Node", </ins> }, }

/kind bug What happened: doesn't show role Workaround: What you expected to happen: According to and How to reproduce it (as minimally and precisely as possible): I just deployed a multi master kubernetes environment with kops: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS Install tools: kops, terraform
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig cli
I don't think this is a bug. The node role label is not used by kubeadm anymore. Related codes have been changed in by
Hi Thanks for the quick reply I'm actually using the as you can see in the workaround above. I'm not using kubeadm role label.
Your favorite topic came up again. Hi I do think this is a bug. As said, kubectl shouldn't show anything based on the old kubeadm master label, but it should probably on the label, but absolutely on the label. cc as you have opinions on this as well cc


Question: import ( "encoding/json" "fmt" <ins> "math" </ins> "net" "os" "path"

Help review this, thanks.
Are you asking for better validation on pod creation? I'm not really sure what you mean when you say "tip".
Yes, I mean it's better validate the pod when create. The tip mean the validation info response from server and show by client.


Question: ```console [12:54:41] $ ./cluster/kubectl.sh describe pod simmemleak-hra99 <del> Name:               simmemleak-hra99 Namespace:          default Image(s):           saadali/simmemleak Node:               kubernetes-minion-tf0f/10.240.216.66 Labels:             name=simmemleak Status:             Running </del> <ins> Name:                           simmemleak-hra99 Namespace:                      default Image(s):                       saadali/simmemleak Node:                           kubernetes-minion-tf0f/10.240.216.66 Labels:                         name=simmemleak Status:                         Running </ins> Reason: Message: <del> IP:             10.244.2.75 Replication Controllers:    simmemleak (1/1 replicas created) </del> <ins> IP:                             10.244.2.75 Replication Controllers:        simmemleak (1/1 replicas created) </ins> Containers: simmemleak: Image:  saadali/simmemleak Limits: <del> cpu:      100m memory:       50Mi State:      Running Started:      Tue, 07 Jul 2015 12:54:41 -0700 Ready:      False Restart Count:  5 </del> <ins> cpu:                      100m memory:                   50Mi State:                      Running Started:                  Tue, 07 Jul 2015 12:54:41 -0700 Last Termination State:     Terminated Exit Code:                1 Started:                  Fri, 07 Jul 2015 12:54:30 -0700 Finished:                 Fri, 07 Jul 2015 12:54:33 -0700 Ready:                      False Restart Count:              5 </ins> Conditions: Type      Status Ready     False

Problem When a container is killed and restarted because, for example, it hit memory limits, the output only shows an increasing but no reason for why the container was killed: We can find the reason by inspecting the pod object ( shows as with ): Suggested Solution Modify to print termination reason, if available.
When this merges, need to update Search for in that doc.
Solved with .


Question: }) /* <del> Testname: var-expansion-subpath-test-writes Description: Verify that a subpath expansion can be used to write files into subpaths. 1.	valid subpathexpr starts a container running 2.	test for valid subpath writes 3.	successful expansion of the subpathexpr isn't required for volume cleanup </del> <ins> Release : v1.19 Testname: VolumeSubpathEnvExpansion, subpath test writes Description: Verify that a subpath expansion can be used to write files into subpaths. 1.	valid subpathexpr starts a container running 2.	test for valid subpath writes 3.	successful expansion of the subpathexpr isn't required for volume cleanup </ins> */ <del> ginkgo.It("should succeed in writing subpaths in container [sig-storage][Slow]", func() { </del> <ins> framework.ConformanceIt("should succeed in writing subpaths in container [sig-storage][Slow]", func() { </ins> podName := "var-expansion-" + string(uuid.NewUUID()) containerName := "dapi-container"

Is this a BUG REPORT or FEATURE REQUEST?: /kind api-change /sig storage /sig node /sig testing Description: Promote e2e tests for VolumeSubpathEnvExpansion to Conformance as it is GA approved. Related Issues: GA Upgrade: Feature Request: KEP:
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: if err != nil { return nil, fmt.Errorf("Error creating loadbalancer %v: %v", createOpts, err) } return loadbalancer, nil } <ins> func stringInArray(x string, list []string) bool { for _, y := range list { if y == x { return true } } return false } </ins> func (lbaas *LbaasV2) GetLoadBalancer(clusterName string, service *api.Service) (*api.LoadBalancerStatus, bool, error) { loadBalancerName := cloudprovider.GetLoadBalancerName(service) loadbalancer, err := getLoadbalancerByName(lbaas.network, loadBalancerName)

Kubernetes version: $ kubectl version Client Version: version.Info{Major:"1", Minor:"3+", GitVersion:"v1.3.8-hpe.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-22T23:27:29Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"3+", GitVersion:"v1.3.8-hpe.1.3+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-09-29T14:43:35Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: OpenStack Liberty, Neutron LBaaS V2 using Octavia OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A What happened: Deleting a LoadBalancer service results in the deletion of all LoadBalancer service's LBaaS Pools and Monitors. What you expected to happen: The Pools and Monitors for unrelated services should not be deleted How to reproduce it (as minimally and precisely as possible): 2 services with type=LoadBalancer for LBaaS resources to be created 1 of the 2 services created earlier output, all pools will have been deleted. Anything else do we need to know: Neutron's API ignores unknown paramaters. When listing pools etc, K8 attempts to filter on "LoadBalancerID", which is not a valid filter. As such, it is ignored by Neutron, and a list of all pools is returned. K8 then proceeds to delete each of the pools.
We're still experiencing this problem, or at least one with very similar symptoms -- having launched services of type LoadBalancer and then deleting one of them, it deletes all of the pools of all services. As the original report, we're also using OpenStack Liberty, Neutron LBaaS V2, and just updated to K8s 1.5.1 (from 1.4.5) hoping that this fixed our issue, but we are still experiencing the same problem after the upgrade.

Is there any other way in which we can help to debug the issue, without delving into the code? (On a side note, the commit which is supposed to fix this is included as part of the 1.5.0 release, so I am not sure if this issue has not been closed for some reason).
it is possible I missed a case in the original fix, the simplest test I found for this issue we had was to create 2x services with type=loadbalancer, wait for all the LBaaS resources to be provisioned, then delete one of the services and ensure the other's resources are still present. If this is failing, either these been a regression, or something else has gone horribly wrong :)
Hi thanks for coming back to this issue. Apparently not all cases deleting the LoadBalancers in OpenStack/liberty were doing the extra step of filtering. We were working on a fix, and submitted a pull request, which is being considered right now:
has merged. closing this issue /close


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kubelet const ( NetworkNotReadyErrorMsg = "network is not ready" ) </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pods are synchronized by kubelet before network is ready. Synchronization in such case fails and pod is synchronized after 10 seconds. What you expected to happen: Kubelet should wait until network is ready before starting synchronizing pods. How to reproduce it (as minimally and precisely as possible): Configure cluster with Cluster Autoscaler enabled. Start pod so cluster is scaled up. In logs you might find following error: E0907 13:18:49. 972 ] Error syncing pod xxxxxx ("xxxxxxxxx(xxxxxxxxxxx)"), skipping: network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] Environment: Build from HEAD Cloud provider or hardware configuration: GKE
/kind bug /sig node


Question: ginkgo.It("should never report success for a pending container", func() { ginkgo.By("creating pods that should always exit 1 and terminating the pod after a random delay") <del> var reBug88766 = regexp.MustCompile(`ContainerCannotRun.*rootfs_linux.go.*kubernetes.io~secret.*no such file or directory`) </del> <ins> var reBug88766 = regexp.MustCompile(`rootfs_linux.*kubernetes.io~secret.*no such file or directory`) </ins> var ( lock sync.Mutex

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: Triage board shows flakes here spiked dramatically on 3/4 /sig node /priority critical-urgent /milestone v1.18
are you able to assist if this is related to recent topology work?
sure, I'll take a look
is this the test you just in ?
As mentioned above, this is a new test that was just , and the test sometimes fails. I don't see that any of the recent topology manager changes would cause these failures. I'll be happy to help out and dig more if or others think otherwise.
This is me, the bugs are untriaged containerd / Kubelet failures with low occurrence rates that this test now exposes. Investigating.
still flaking in non-dockershim environments: pod pod-submit-status-0-14 on node kind-worker2 container unexpected exit code 128: start=1970-01-01 00:00:00 +0000 UTC end=2020-03-06 16:16:54 +0000 UTC reason=StartError message=failed to create containerd task: OCI runtime create failed: starting container process caused " container init caused " mounting "/var/lib/kubelet/pods/-819e-4840-8369-" to rootfs "" at "" caused "stat /var/lib/kubelet/pods/-819e-4840-8369- no such file or directory""": unknown `
Different reason this time. Will continue to enumerate failure modes.


Question: } else { klog.V(4).Infof("running pod %q is pending termination", podPair.RunningPod.ID) } <del> return true </del> <ins> return </ins> } <del> return false </del> <ins> // Limit to one request per pod pk.podKillingCh <- podPair </ins> } // Close closes the channel through which requests are delivered

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Right after calling , returns false even the pod is not killed yet. What you expected to happen: In kubelet, after is complete, the pod should be pending termination or be killed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Related to Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /assign


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> cloudprovider "k8s.io/cloud-provider" cloudproviderapi "k8s.io/cloud-provider/api" "k8s.io/klog"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files.

I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?


Question: } } sortObjects(assumedClaims) <del> stripObjects(assumedClaims) </del> return assumedClaims }

pull-kubernetes-unit dynamicresources unknown multiple failures in tests as seen in n/a /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably cc
/assign
DeepCopy missing here:


Question: t.Fatalf("Expected %#v, Got %#v", expected, update) } <del> case <-time.After(2 * time.Millisecond): </del> <ins> case <-time.After(time.Second): </ins> t.Errorf("Expected update, timeout instead") } }

W0128 18:44:26. ] Pod from test failed validation, ignoring: namespace: required value '' E0128 18:44:26. ] Could not construct reference to: '&api.BoundPod{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"foo", Namespace:"", SelfLink:"", UID:"foo", ResourceVersion:"", CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0x0, loc:(time.Location)(nil)}}, Labels:map[string]string(nil), Annotations:map[string]string{"":""}}, Spec:api.PodSpec{Volumes:[]api.Volume(nil), Containers:[]api.Container(nil), RestartPolicy:api.RestartPolicy{Always:(api.RestartPolicyAlways)(0xe3d670), OnFailure:(api.RestartPolicyOnFailure)(nil), Never:(api.RestartPolicyNever)(nil)}, DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), Host:""}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'failedValidation' 'Error validating pod foo.<empty-namespacefrom test, ignoring: namespace: required value E0128 18:44:28. ] Unable to read config path "randomnonexistentpath": path does not exist, ignoring --- FAIL: TestReadFromFileWithoutID (0.20 seconds) Expected update, timeout instead


Question: } } <ins> // TestLoadConfig tests proper operation of loadConfig() </ins> func TestLoadConfig(t *testing.T) { <del> yaml := `apiVersion: componentconfig/v1alpha1 bindAddress: 9.8.7.6 </del> <ins> yamlTemplate := `apiVersion: componentconfig/v1alpha1 bindAddress: %s </ins> clientConnection: acceptContentTypes: "abc" burst: 100 contentType: content-type kubeconfig: "/path/to/kubeconfig" qps: 7 <del> clusterCIDR: "1.2.3.0/24" </del> <ins> clusterCIDR: "%s" </ins> configSyncPeriod: 15s conntrack: max: 4

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? is:issue is:open ipv6 Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): N/A, these are unit test changes in latest master Environment: Cloud provider or hardware configuration: Kubernetes container dev env on Ubuntu 16.04.1 OS (e.g. from /etc/os-release): Ubuntu 16.04.1 Kernel (e.g. ): 4.4.0-78-generic -Ubuntu x8664 GNU/Linux Install tools: Others: What happened: Upon inspection, no IPv6 test cases found in kubernetes/cmd/kube- What you expected to happen: Expect to see some IPv6 test cases exercised for kube-proxy server code (kubernetes/cmd/kube-). Also need some code coverage for negative conditions in How to reproduce it (as minimally and precisely as possible): Run unit tests in kubernetes/cmd/kube- Anything else we need to know:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig networking
/area ipv6
/sig network


Question: awsServices.ec2.(*MockedFakeEC2).AssertExpectations(t) } <ins> func TestGetCandidateZonesForDynamicVolume(t *testing.T) { tests := []struct { name          string labels        map[string]string ready         bool expectedZones sets.String }{ { name:          "master node with role label", labels:        map[string]string{labelKeyNodeRole: nodeMasterRole, v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString(), }, { name:          "master node with master label empty", labels:        map[string]string{labelKeyNodeMaster: "", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString(), }, { name:          "master node with master label true", labels:        map[string]string{labelKeyNodeMaster: "true", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString(), }, { name:          "master node with master label false", labels:        map[string]string{labelKeyNodeMaster: "false", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "minion node with role label", labels:        map[string]string{labelKeyNodeRole: nodeMinionRole, v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "minion node with minion label", labels:        map[string]string{labelKeyNodeMinion: "", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "minion node with compute label", labels:        map[string]string{labelKeyNodeCompute: "true", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "master and minion node", labels:        map[string]string{labelKeyNodeMaster: "true", labelKeyNodeCompute: "true", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "node not ready", labels:        map[string]string{v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         false, expectedZones: sets.NewString(), }, { name:          "node has no zone", labels:        map[string]string{}, ready:         true, expectedZones: sets.NewString(), }, { name:          "node with no label", labels:        map[string]string{v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, } for i, test := range tests { t.Run(test.name, func(t *testing.T) { awsServices := newMockedFakeAWSServices(TestClusterID) c, _ := newAWSCloud(CloudConfig{}, awsServices) c.kubeClient = fake.NewSimpleClientset() nodeName := fmt.Sprintf("node-%d", i) _, err := c.kubeClient.CoreV1().Nodes().Create(&v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:   nodeName, Labels: test.labels, }, Status: genNodeStatus(test.ready), }) assert.Nil(t, err) zones, err := c.GetCandidateZonesForDynamicVolume() assert.Nil(t, err) assert.Equal(t, test.expectedZones, zones) }) } } func genNodeStatus(ready bool) v1.NodeStatus { status := v1.ConditionFalse if ready { status = v1.ConditionTrue } ret := v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:   v1.NodeReady, Status: status, }, }, } return ret } </ins> func newMockedFakeAWSServices(id string) *FakeAWSServices { s := NewFakeAWSServices(id) s.ec2 = &MockedFakeEC2{FakeEC2Impl: s.ec2.(*FakeEC2Impl)}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have pretty large clusters in aws and in production, we found an inefficiency in provisioning dynamic EBS volumes, where aws-ebs volume provisioner tries to look for available zone from aws directly. With scale, this can trigger massive from ec2, and affect cluster stability. This is also related to What you expected to happen: To simply get zones with available instances, we can get information from API server for efficiency. There might be a delay between node gets shutting down and such change is reflected in api server, but in case node is not ready or gets terminated, there will be higher level reconciliation triggered so we are safe. How to reproduce it (as minimally and precisely as possible): We only expect this to be hit when the cluster is huge (or aws account is huge) so not easy to reproduce. /assign /sig aws /sig cloud-provider
/assign


Question: continue } if volume.pluginIsAttachable { <del> err = rc.actualStateOfWorld.MarkDeviceAsMounted(volume.volumeName) </del> <ins> err = rc.actualStateOfWorld.MarkDeviceAsMounted(volume.volumeName, volume.devicePath) </ins> if err != nil { glog.Errorf("Could not mark device is mounted to actual state of world: %v", err) continue

We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by


Question: expectVal: defaultIP, }, } <del> for _, c := range cases { </del> <ins> for _, tc := range testCases { </ins> fs := pflag.NewFlagSet("blah", pflag.PanicOnError) ip := defaultIP fs.Var(IPVar{&ip}, "ip", "the ip")

Whenever I try to change kube proxy metrics bind using flag , it gives me: Then, I the port , the error changes. I expect the metrics server to listen to 0.0.0.0:. Do I miss something? Environment: Kubernetes version: v1.7.8 Cloud provider or hardware configuration**: on-premise OS (e.g. from /etc/os-release): debian/stretch Kernel (e.g. ): Linux R11U2-7U20N-1 4.9.0-3-amd64 SMP Debian 4.9.30-2+deb9u3 (2017-08-06) x86_64 GNU/Linux Install tools: Others:
/sig network /area kube-proxy
The is being and does NOT allow specifying a port. Have you tried using a config file? The field allows you to specify a port (an example ).
/assign
Turned out we use a wrong type when validating and . Sent for fixing this. Will try to cherrypick the fix as well.
New finding, is broken on HEAD in a different way: Seems to be introduced by
Humm..We have some default logic for : Seems like for some reason it does not auto append the default port number OR the validation happens before defaulting.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
I got the same issue 0.0.0.0 0.0.0.0: both not work with kube-proxy 1.9.3 amd64
It is a BUG. Conflict between componentconfig.IPVar.Set(s string) and validation.validateHostPort.

Could you elaborate the issue a bit more? Which k8s version are you on?
I also meet the same issue with kubernetes 1.8. It's impossibe for Prometheus to fetch metrics from 127.0.0.1: in my case, any workaround? I think we'd better reopen this issue.
metrics binds to by default, are you trying to modify port as well? Thanks for reporting, I confirm the issue is remaining. (or any ip:port) does not work as intended even after the fix. Reopening this and filed a new issue for conflict kubeproxyconfig validation. /reopen
Thanks for reopening the issue. I want to bind metrics to 0.0.0.0: as well. With address 127.0.0.1:, Prometheus cannot pull metrics from this address unless they are on the same node.
is fixing this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale (I just got bitten by this on kube-proxy v1.9.10)
should fix this.
Closing this as is merged. /close
Closing this issue. <detailsIn response to : Closing this as is merged. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } else { predicateResults[predicateKey] = HostPredicate{Fit: fit, FailReasons: reasons} } <ins> result := predicateResults[predicateKey] ecache.UpdateCachedPredicateItem(pod.GetName(), info.Node().GetName(), predicateKey, result.Fit, result.FailReasons, equivCacheInfo.hash, false) </ins> } } <ins> if eCacheAvailable { ecache.Unlock() } </ins> if !fit { // eCache is available and valid, and predicates result is unfit, record the fail reasons failedPredicates = append(failedPredicates, reasons...)

Fix this TODO (raised by : Part of: One thoughts to fix this is, we can redesign the equivalence cache data structure to: bool instead of deleting the entry. certain version tag in the entry, update the version per WRITE op (Update and Invalidate). the version after eCache predicate, and check it before update eCache.
Another possible approach is: eCache before running a predicate. the predicate. the eCache for the predicate. eCache. Basically, in the existing code all predicates are run first and their results are collected and then at the end of the function the eCache is updated. Instead of that, we can move the eCache update for each predicate inside the for loop and next to each predicate execution and synchronize before start of the execution of each predicate. The downside to my suggested approach is that performance may suffer due to frequent locking and unlocking.
Prefer to bsalamat@ 's option which is simpler :). Comparing to performance improvement by eCache, the lock is acceptable to me.
yup same here, going with the locking is easier and is acceptable. - also using a flag for invalidation might just move the problem from eCache to the flag ?
Oh I missed this notification somehow, busy on cleaning scheduler integration test. Yep so if locking is acceptable, it should be cleanest way to fix this, optimistic control would be a more complicated story.
- are you handling this one ?
Already been taken by volunteer days ago :)


Question: ns := f.Namespace var err error <ins> // TODO: This skip should be removed once #70760 is fixed skipTestUntilBugfix("70760", input.driverName, []string{"csi-hostpath", "com.google.csi.gcepd"}) </ins> By("Creating sc") input.sc, err = cs.StorageV1().StorageClasses().Create(input.sc) Expect(err).NotTo(HaveOccurred())

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Probably caused by <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
https://k8s- /assign
I understand that moved crd creation logic to e2e test's k8s cluster deployment code. On the other hand, I tested it with manually deployed k8s cluster. So, in my case, I guess that in addition to enable and feature gates enabled, I need to create crd manually before tests, like below: I could make two of three SkipAttach tests pass in my environment by above way. For "non-attachable volume does not need VolumeAttachment" case, I'm still looking into it. (It doesn't seem to skip creating volumeAttachment, properly)
I found a typo in 's argument. While hostpath driver's connects driver name and unique name without "-", it connects them with "-". This makes "non-attachable" test fail, due to the mismatch between volume name in crd and provisioner name, as a result, creating volumeAttachment isn't skipped. I will create a PR to fix it. For above bug, is unrelated. It just changed the way in how to test in manually deployed k8s cluster as I mentioned in above comment. So, there should be no problem for the PR, at least for the CI testing if it works well in CI environments.
I can verify that the CRDs are not getting installed on the alpha cluster. Investigating why.
/reopen since I see the Alpha tests failing
/milestone v1.13 /priority important-soon /kind failing-tests
/kind failing-test
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is fully resolved. There are still failures on https://k8s- /reopen
Reopened this issue. <detailsIn response to : I don't think this is fully resolved. There are still failures on https://k8s- Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you help take a look at these failures? It's testing CSI filesystem volume mode. I think it is unrelated to the crd installation issue that Saad fixed.
I tested with AllAlpha flag and manually created csi crds and confirmed that filesystem volume mode test passed in my environment and also other tests passed there. If I delete csi crds, other tests also fail, so I agree that the issue that Saad fixed is not related. However, above also means that this issue can't be reproduced in my environment.
I1109 14:56:01.141] Nov 9 14:56:01.131: INFO: At 2018-11-09 14:50:54 +0000 UTC - event for security-context--e42e-11e8-9b95-: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc--e42e-11e8-880b-" : node "bootstrap-e2e-minion-group-3wtg" has no NodeID annotation
Hostpath also seems to be failing because of driver registration issues? Not sure why other alpha tests are passing if that's the case
This test is the only test that uses  to create pod with CSI driver volume and expects the result to be success. One of the pod parameters specified there might affect this failure.
Those 2 failing tests are newly introduced in and started to fail from the beginning. Is that PR blocking 1.13? If not, can we revert and fix later? I have recently set up optional presubmit tests for alpha features at , which will pick up those 2 tests. Currently this failure is blocking
apart from the 2 failing tests, I see which addressed with is also flaky. These failures are now blocking PRs as mentioned above. As we enter code slush today, its very critical to have the PRs get in sooner than later. Can we plz revert the feature and the tests until they are fully resolved and ready to go. Thanks
I prefer disabling the specific tests instead of reverting the entire pr. We will lose important signal for csi
That works for me.

Can you cc me on the PR that disables those tests? When you re-enable the tests, remember to try them out on . :)
Let me consider if we can just skip these test for CSI, instead of reverting the entire patch.
you mean just skipping them in Alpha cluster right?
I was thinking about skipping the below test with the combination of below drivers: [Test] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode[Feature:BlockVolume] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources [Driver] csi-hostpath Won't it enough? Or, do we need to skip it only with alpha cluster?
If we just disable the tests alone and not revert the feature, we are just masking the issue. Ideally if we are not able to fix the tests and reenable them the feature should be reverted from 1.13 as well. Are you also actively working on fixing the tests and this skip is just to unblock PRs in the meantime? if so can we timebox by when we can have the tests fixed and re-enabled / the feature reverted? My fear is we might forget the fact that its disabled since there's no active reporting on it.
Actually, there is no feature in either of culprit commits. It tests or intended to improve the way it is tested. Therefore, it must be a test issue or another issue which is found by the test , not an issue introduced by the commits themselves. In the code, there is a clear comment that the test is skipped due to this issue with the reference to this issue#, so this won't be forgot. If you still worry about this issue is not tracked, we might be able to keep this issue open.
IIUC the test that were failing (and are being reverted) were for the new CSI festures going to Alpha in 1.13, right? so even these 2 culprit commits do not contain feature work, it is testing some new code additions to 1.13 and now those will be untested until we re-enable these tests. Is there an ETA to re-enable them, preferably sometime early next week? Yes +1 to keeping this issue open to track the skipped tests.
The two failing tests themselves are not for new alpha features that are in v1.13.

Instead, they are just to test csi driver works well as it works in existing in-tree driver. (And failed because they might be affected by some other alpha features or some of specific configurations in gce.) In addition, ~50 tests for csi driver, so it would be better not to remove all of the tests by reverting the PR, just for the 2 failing tests. I think that just reverting the patch won't help improving quality of k8s. As for ETA to solve the issue, I found difficulty in debuging in my local environment. It seems that the two tests fail in gce environment w/ AllAlpha=true, however, as I've already mentioned, at least csi-hostpath tests in my local environment w/ AllAlpha=true doesn't fail.(And I don't have test environment for gce.) Of course, I will help solving the issue next week. However, I will need someone's help for testing combinations of alpha features in gce environment to narrow down which flag really affects this issue. At least I would like to know the results: w/ MountPropagation=true,CSIPersistentVolume=true,BlockVolume=true,CSIBlockVolume=true all alpha flags except CSIDriverRegistry and CSINodeInfo
/reopen Thanks for the detailed explanation. I think the revert is fine in this case then. I am keeping this issue open however to track and fix the failures in these 2 tests can any of you help answer w.r.t alpha features on gce environment. thanks
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can help debug when I'm back on Tuesday. In the meantime, I would suggest looking through the failing test logs to try to triage what's going on
Here's one failing run:
I looked into the csi-hostpath failure log. As seen in the for , where the pod which has a csi-hostpath volume is being attached is scheduled, mount volume fails with these errors (1). Related code to this log is (2), so the driver doesn't seem to be registered to . As for the registration code for csiDriver, it is called as (3).

Therefore, will be logged if the driver is triggered for registration. As seen (4), registration is only triggered in , which is different from the one the pod is scheduled. In addition, driver-registrar seems to be deployed only on the node where registration is triggered (5). Is it right behaviour, while registrar is deployed as daemonset? I guess that just this fact seems to cause an issue to make csi-hostpath driver work properly. (Actually, I only tested with 1 node environment, so this might only happen in multiple nodes environment.) <details- (1) (2) Result of grep to check whether registrar is deployed to each node [ for bootstrap-e2e-minion-group-50r7][ for bootstrap-e2e-minion-group-k781][ for bootstrap-e2e-minion-group-z5cg] </details
The hostpath plugin should be deployed as a DaemonSet, so it's strange that it's only getting registered on only one node.
/assign looking into this as well
could you link the PR for skipping [external-gcepd] tests into this issue when you do that as well
skipping gcePD-external is here:
Since gce-cos-alphafeatures is now passing, do we need to keep this issue open? Should it morph into a non-blocking issue for longterm work on a better test?
We have root caused 2 of the 3 failures, which were problems in the test and test conifg. Fixes will be out soon. Still triaging the 3rd failure, which could be a real bug in alpha clusters
I've root caused csi-hostpath failure issue. It just failed to set the node properly in test code side. I will make a PR to fix it.
Or should it be done in your PR too, to avoid modifying twice in the separate PR? Actually, the fix are as below: (Failing to set node before pod creation)
Created for csi-hostpath fix, to prepare for the case that we need a separate PR. We can discuss more on this there, if needed.
/unassign found possible root cause for GCE issue: CSI NodeID annotation is not present on the Node object, could be caused by failed retries on updating Node object.

(apparently when alpha features are open we shouldn't be looking at the Node object anyway except we seem to be, so that might also be a bug). has the most context for the fix /assign
The symptom of the remaining test failure is the test pod fails to create because it fails to attach with the error: . The test succeeds the first time it's run in a cluster, and always fails after that. Two issues caused this error message. the CSI external-attacher tries to fetch the object to get the NodeID, it does so through an informer. The first time the test is run, objects are created for the very first time, and the informer returns the correct object. In subsequent test runs, s from the first run is kept, and for some reason the informer fails to detect the object and returns empty. Making the API call directly works, I'll patch the external-provisioner with this to mitigate the issue, and this is sufficient to get the test to pass. The informer bug will need to be eventually fixed. fails to post the NodeID annotation sometimes, likely due to node status patch not being retried correctly. This is crucial for clusters without feature flag set, and this is the default for 1.13. Will add the appropriate retry mechanism.
Opened to track the informer issue
Are and potential PR for the only 2 fixes being actively worked on for this issue?
Only is needed. We have a workaround for
will you have a followup PR to enable the CSI alpha tests back? or is that already in flight?
also reenables the tests
OK with merged we should see the CSI alpha tests re-enable. to keep an eye out for CI signal
The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! /close
Closing this issue. <detailsIn response to : The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The tests are actually running (and passing) in the normal suite now, now that the raw block feature moved to beta.


Question: p = path.Join(p, r.resourceName, r.subresource, r.subpath) } <del> finalURL := *r.baseURL </del> <ins> finalURL := url.URL{} if r.baseURL != nil { finalURL = *r.baseURL } </ins> finalURL.Path = p query := url.Values{}

Currently we do not measure latency for REST operations. This is important for any component that performs non-watch operations (scheduler, controllers, kubelet). This is also prerequisite to actually make sure that all API calls take less than 1 sec ( ).
The apiserver is already , including for REST operations. You can scrape the metrics by hitting the /metrics HTTP endpoint.
Sorry, I was not clear in the description. I'm more interested on the client side and with more details so that I can measure things like: how long it takes to write bindings in scheduler how long it takes to update status in kubelet I'd think about it as a counterpart for apiserver stats.


Question: // Check against i+1 so we exit if minTries == maxTries. if (responses.Equal(expectedResponses) || responses.Len() == 0 && expectedResponses.Len() == 0) && i+1 >= minTries { <del> return </del> <ins> framework.Logf("reached %v after %v/%v tries", targetIP, i, maxTries) return nil </ins> } // TODO: get rid of this delay #36281 time.Sleep(hitEndpointRetryDelay) } if dialCommand == echoHostname { config.diagnoseMissingEndpoints(responses) } <del> framework.Failf("Failed to find expected responses:nTries %dnCommand %vnretrieved %vnexpected %vn", maxTries, cmd, responses, expectedResponses) </del> <ins> returnMsg := fmt.Errorf("did not find expected responses... nTries %dnCommand %vnretrieved %vnexpected %v", maxTries, cmd, responses, expectedResponses) framework.Logf("encountered error during dial (%v)", returnMsg) return returnMsg </ins> } // GetEndpointsFromTestContainer executes a curl via kubectl exec in a test container.

What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: assert.NotNil(c8.Memory.Time) checkCRIPodCPUAndMemoryStats(assert, p3, infos[sandbox3Cgroup].Stats[0]) <ins> p6 := podStatsMap[statsapi.PodReference{Name: "sandbox6-name", UID: "sandbox6-uid", Namespace: "sandbox6-ns"}] assert.Equal(sandbox6.CreatedAt, p6.StartTime.UnixNano()) assert.Equal(1, len(p6.Containers)) c9 := p6.Containers[0] assert.Equal(cName9, c9.Name) assert.Equal(container9.CreatedAt, c9.StartTime.UnixNano()) assert.NotNil(c9.CPU.Time) assert.Equal(containerStats9.Cpu.Timestamp, p6.CPU.Time.UnixNano()) assert.NotNil(c9.Memory.Time) assert.Equal(containerStats9.Memory.Timestamp, p6.Memory.Time.UnixNano()) </ins> mockCadvisor.AssertExpectations(t) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: CPU.Time is set to cs.StartTime but ps.Memory.Time is set to timestamp when collecting the memoryusage. What you expected to happen: CPU.Time is set to cs.CPU.Time How to reproduce it (as minimally and precisely as possible): Check the code Anything else we need to know?: Environment: Kubernetes version (use ): master branch when creating this issue. Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig kubelet
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig kubelet Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I am happy to take over this. /assign


Question: PersistentVolumeReclaimPolicy: c.options.PersistentVolumeReclaimPolicy, AccessModes:                   c.options.PVC.Spec.AccessModes, Capacity: v1.ResourceList{ <del> v1.ResourceName(v1.ResourceStorage): resource.MustParse(fmt.Sprintf("%dGi", sizeGB)), </del> <ins> v1.ResourceName(v1.ResourceStorage): resource.MustParse(fmt.Sprintf("%dG", sizeGB)), </ins> }, PersistentVolumeSource: v1.PersistentVolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{

GCE PDs are allocated in chunks of GBs (not GiB) but function incorrectly creates volume in chunks of GiB - cc /sig storage
/kind bug
/assign
/priority important-soon
Hey I submit a pr for this issue. Your suggestions are very important. Thanks.


Question: kubeClient:                              kubeDeps.KubeClient, heartbeatClient:                         kubeDeps.HeartbeatClient, onRepeatedHeartbeatFailure:              kubeDeps.OnHeartbeatFailure, <del> rootDirectory:                           rootDirectory, </del> <ins> rootDirectory:                           filepath.Clean(rootDirectory), </ins> resyncInterval:                          kubeCfg.SyncFrequency.Duration, sourcesReady:                            config.NewSourcesReady(kubeDeps.PodConfig.SeenAllSources), registerNode:                            registerNode,

pull-kubernetes-unit, see example failure in It appears to be a new test in No response No response No response /sig
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Repro's locally:
/cc can you PTAL? It seems related to new test in
flaking regularly -
actually, I'm not sure this is a test-only race... it looks like self mutates , and that is called in a goroutine in parallel with other methods that access rootDirectory: I suspect this has always been a race, and the unit test is exercising / catching it more
moving the filepath.Clean to init time: and changing the check in setupDataDirs to be a validation rather than a mutation: might resolve the immediate race
/assign
is open to fixing it. The data race we find here should be fixed. However, is another one that we need to fix. I add a new commit to fix that as well.
did you see that race as well or did you just see the issue opened? that was in a code path that should be feature gated so I wouldn't have expected it to trigger in the kubemark unit tests
I may mixed problems together. Yes. will not trigger in the kubemark unit tests. Using stressing, I find the data race cannot fix the problem completely. Then I saw and cherry-picked the fix to my PR. But the hollowNode test still flakes and I looked into it and find it is the volumeManager problem. is just another potential data race in .

the rootDirectory and volumeManagerSyncTime race fixes are easy to reason about and fix code that looks like it could really race in currently enabled-by-default code looks like a real race in disabled-by-default code. I'll defer to kubelet folks and whether to mix that fix into
I think I should drop that fix. This will make things more clear.


Question: podTemplate.Labels = jig.Labels pod, err := cs.CoreV1().Pods(ns).Create(podTemplate) framework.ExpectNoError(err) <ins> framework.ExpectNoError(f.WaitForPodReady(pod.Name)) </ins> ginkgo.By("waiting for the service to expose an endpoint") err = e2eendpoints.ValidateEndpointsPorts(cs, ns, serviceName, e2eendpoints.PortsByPodName{serverPodName: {servicePort}})

Per PR the test fails to error properly. One line of particular interest from that PR: If you wonder why it does not fail on docker, that's because the test does not report the error correctly but fails the same way. This issue is intended to track the discussion and resolution of making this test error properly so that it is useful to CI Signal /sig networking /cc
The label(s) cannot be appled. These labels are supported: <detailsIn response to : Per PR the test fails to error properly. One line of particular interest from that PR: If you wonder why it does not fail on docker, that's because the test does not report the error correctly but fails the same way. ` /cc </details
I'm following up on this in
/sig network


Question: framework.ExpectNoError(err) ginkgo.By("Verifying the LimitRange was deleted") <del> gomega.Expect(wait.Poll(time.Second*5, e2eservice.RespondingTimeout, func() (bool, error) { selector := labels.SelectorFromSet(labels.Set(map[string]string{"name": limitRange.Name})) options := metav1.ListOptions{LabelSelector: selector.String()} limitRanges, err := f.ClientSet.CoreV1().LimitRanges(f.Namespace.Name).List(context.TODO(), options) </del> <ins> err = wait.Poll(time.Second*5, e2eservice.RespondingTimeout, func() (bool, error) { limitRanges, err := f.ClientSet.CoreV1().LimitRanges(f.Namespace.Name).List(context.TODO(), metav1.ListOptions{}) </ins> if err != nil { framework.Logf("Unable to retrieve LimitRanges: %v", err)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /priority critical-urgent /cc /milestone v1.19 This test waits for the limit range to be deleted before attempting to create the so this error is surprising. It appears there are no limit ranges present when they are listed here:
It seems this job failed just once since Jul 23, so I'm wondering is it a must for 1.19?
Given this is a conformance test I would prefer we at least know what is causing the issue
This is still flaking with this same issue. /reopen
Reopened this issue. <detailsIn response to : This is still flaking with this same issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.20
This issue has not occurred recently and seems to have already been fixed. I think we can close this issue.
Yes, was expected to get it fixed. /close
Closing this issue. <detailsIn response to : Yes, was expected to get it fixed. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this is still flaking in the 1.20 branch : confused : as of one hour ago


Question: pkg: - installed <del> {% set logrotate_files = ['kube-scheduler', 'kube-proxy', 'kubelet', 'kube-apiserver', 'kube-controller-manager', 'kube-addons'] %} </del> <ins> {% set logrotate_files = ['kube-scheduler', 'kube-proxy', 'kubelet', 'kube-apiserver', 'kube-controller-manager', 'kube-addons', 'docker'] %} </ins> {% for file in logrotate_files %} /etc/logrotate.d/{{ file }}: file:

We have rotation of logs from docker containers, but not for logs from docker Without rotation, the logs may fill the root partition entirely.
I can add rotation (5 files, 100MB each) if you decide it goes for v1.0 (for me it's a blocker). I'm not sure which platforms are affected. For sure GCE. cc
Sure, send a PR and I'm sure we can merge it.


Question: test.nodeName, test.predicateKey, test.equivalenceHashForCalPredicate, <ins> true, </ins> ) // returned invalid should match expectedInvalidPredicateKey or expectedInvalidEquivalenceHash if test.equivalenceHashForUpdatePredicate != test.equivalenceHashForCalPredicate {

Fix this TODO (raised by : Part of: One thoughts to fix this is, we can redesign the equivalence cache data structure to: bool instead of deleting the entry. certain version tag in the entry, update the version per WRITE op (Update and Invalidate). the version after eCache predicate, and check it before update eCache.
Another possible approach is: eCache before running a predicate. the predicate. the eCache for the predicate. eCache. Basically, in the existing code all predicates are run first and their results are collected and then at the end of the function the eCache is updated. Instead of that, we can move the eCache update for each predicate inside the for loop and next to each predicate execution and synchronize before start of the execution of each predicate. The downside to my suggested approach is that performance may suffer due to frequent locking and unlocking.
Prefer to bsalamat@ 's option which is simpler :). Comparing to performance improvement by eCache, the lock is acceptable to me.
yup same here, going with the locking is easier and is acceptable. - also using a flag for invalidation might just move the problem from eCache to the flag ?
Oh I missed this notification somehow, busy on cleaning scheduler integration test. Yep so if locking is acceptable, it should be cleanest way to fix this, optimistic control would be a more complicated story.
- are you handling this one ?
Already been taken by volunteer days ago :)