Question: c.registeredPods[key] = pod if prev != nil { for name := range getSecretNames(prev) { <ins> // On an update, the .Add() call above will have re-incremented the // ref count of any existing secrets, so any secrets that are in both // names and prev need to have their ref counts decremented. Any that // are only in prev need to be completely removed. This unconditional // call takes care of both cases. </ins> c.secretStore.Delete(prev.Namespace, name) } }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Occasionally (more frequently if the node is heavily loaded) the node fails to mount a secret or configmap for a pod. Error recovery fails and the pod must be replaced with a new pod to have any hope of getting the secret or configmap volume mounted. The problem appears to be that if a pod is updated, its secrets are removed from the secretStore's secrets map. See: Normally the effects of the bug are not felt since the usual order of operations masks the bug: 1) The pod is created and registered the first time with the secrets manager 2) The volume mounter gets the secret 3) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) If the race is lost, though, then the bug symptoms are manifest: 1) The pod is created and registered the first time with the secrets manager 2) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) 3) The volume mounter tries to get the secret and fails with the message What you expected to happen: On update, the secrets manager should only delete secrets if they are not present in the latest pod definition (rather than deleting all prior secrets). Whether the pod update happens before or after the volume mounter attempts to get the secret, it should succeed. How to reproduce it (as minimally and precisely as possible): Because it is a race condition, it's difficult to reproduce. We've most often seen the bug on systems that were under load. 1) Create a pod that has a secrets or configmap volume 2) See if the pod fails to mount the volume. Anything else we need to know?: If the above analysis is correct, this bug has been present since 1.6 for secret volumes, and 1.7 for configmap volumes. Environment: We've seen the bug in software based upon the 1.6 release.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
The PR addresses this issue. /sig node
This may be a false alarm. I've never actually reproduced it in Kubernetes proper, only in OpenShift. Since our smoking gun was anything but, I'm going to close this until I manage to find an actual problem in Kubernetes.
To close the loop on this for any that find it via future searches: It turns out that this was a false alarm. The above analysis was flawed and this turns out not to be a bug (or maybe a very minor one if we consider it a bug to log a an error on a (semantic) non-error). The error messages like were not an indicator of a failure to mount a secret volume, but instead a failure to update a secret volume during pod teardown. Here's how the error message is triggered: pod is created volume manager gets the secret and mounts the volume the lifetime of the pod, the volume manager periodically re-checks the secret to make sure that it's still in a good state. pod goes away volume is unmounted If one of the refreshes from step 3 happens after step 4 but before step 5, the log message will appear.


Question: }) // Expect Endpoints resource to be created. <del> if err := wait.PollImmediate(2*time.Second, 12*time.Second, func() (bool, error) { </del> <ins> if err := wait.PollImmediate(2*time.Second, wait.ForeverTestTimeout, func() (bool, error) { </ins> _, err := cs.CoreV1().Endpoints(svc.Namespace).Get(context.TODO(), svc.Name, metav1.GetOptions{}) if err != nil { return false, nil

Which jobs are flaking: ci-kubernetes-kind-e2e-parallel pr:pull-kubernetes-e2e-gce pr:pull-kubernetes-e2e-gce-ubuntu-containerd pr:pull-kubernetes-e2e-gce-ubuntu-containerd ci-kubernetes-kind-ipv6-e2e-parallel Which test(s) are flaking: Second flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know: /sig network /assign
needs investigation since we're switching kube-proxy to endpointslice in this release
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
made me think of , but that's in already.
It's possible it's a test timeout that's just too close, but we need verification of that checking the controller and apiserver logs to make sure there isn't a latent bug
Thanks for catching this! It looks like the timeouts are actually associated with waiting for 2 pods to be ready, not actually anything related to EndpointSlice functionality - timeout is here: Given that, it seems like the obvious solution would be to update the Services to set to and update this block to wait for the Pods to have IPs, not for them to be ready. If that sounds reasonable, I'll work on getting a fix in tomorrow.
/remove-triage unresolved
If that is still exercises the same conditions (I vaguely remember publishNotReadyAddresses changing code paths in the endpoints controller... does it do the same in the endpointslice controller?), that seems reasonable to me.
I think the code path is very similar here, the only real difference should be how the EndpointSlice controller calculates readiness. I've gone ahead and filed a PR that uses this approach, hopefully it helps:
This flaked again this morning /reopen
Reopened this issue. <detailsIn response to : This flaked again this morning Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This should be /priority critical-urgent given
The top level error message does not make this clear but the issue here is that too many EndpointSlices are being created - 3 instead of 2.
Interesting... that condition should not persist for O(minutes), right?

that seems like it might be a bug, not just a test issue
There's potential that it's not consistently merging 2 ports with 1 endpoint into a single slice and instead creating a slice per port. It also could represent a larger bug. Trying to find a way to consistently reproduce this.
Despite my best efforts I have been unable to reproduce this on my own e2e cluster. I've had 10 test runs going at the same time repeatedly without any luck, any ideas for how to reproduce flaky test failures local would be welcome. In the meantime, I've created a follow up PR that will provide more helpful information if/when this test fails again: My leading theory right now is that this could be a caching issue. The EndpointSlice controller runs a sync, sees that we don't have an EndpointSlice for the given port combination in cache and then goes ahead and creates one, even though a matching EndpointSlice already exists and has not been cached yet. That would result in an annoying situation where an essentially duplicate EndpointSlice was created due to a stale cache. I would think the EndpointSliceTracker would resolve this issue for us though, so not sure. Getting the above PR in will help validate this theory. It's worth noting that if this is actually a duplicate EndpointSlice, clients like kube-proxy will handle this seamlessly because they already have to handle potential duplication in EndpointSlices. That would potentially explain why a test failure like this does not seem to be causing other things to break.
It appears that my theory was correct as witnessed here: I create a .

The key points I've noticed so far: 1 EndpointSlice references pod2 (example-named-port-f7vq5) 2 EndpointSlices reference pod1 (example-named-port-fkgcc and example-named-port-vfp2q) 2 EndpointSlices were created at "2020-07-28T04:57:51Z": example-named-port-f7vq5 (references pod2) example-named-port-vfp2q (references pod1) 1 EndpointSlice was created 1 second later ("2020-07-28T04:57:52Z") example-named-port-fkgcc (references pod1) All 3 EndpointSlices are: On generation 1 - no updates have occurred With a state of ready: true The 2 EndpointSlices referencing pod1 are effectively identical  Of course this is something the EndpointSliceTracker should be handling for us, but even that approach seems less than ideal here. It wouldn't actually prevent the duplicate EndpointSlices from being created, it should just be cleaning up duplicates after they happen. This combined with the race conditions referenced in make me think we may want to find a way to slow down the sync cycle for each Service here. This also feels at least tangentially related to the need for better rate limiting in both this and the Endpoints controller discussed in . /cc
Hmm - actually I don't understand why this is happening. Maybe I'm missing something, but I don't think slowing down controller is needed to solve this problem. Even though the controller usually is not lagging my more than 1s, it is possible that it can be lagging by a minute or more in pathological cases. So we can't just rely on the fact that it's guaranteed to be synced. Can you explain it a bit deeper why this is happening? i.e. some more detailed timeline like: T1: state of cache is XYZ T2: controller computes the ES x1, x2 T3: state of cache in ABC T4: controller computes ES y1 [why?]
Agree, I'm not in favor of slowing down the controller globally to address this. Delaying resync of a particular service until we observe the existence of new endpointslices we know we created (with some max delay) could be reasonable.

That's exactly where I was hoping to end up here, although I'm not quite sure how to do that cleanly yet. To be clear, this is not just to fix this flaky test. This flaky test is a symptom of a much larger and more annoying issue with the current implementation. The EndpointSlice controller is consistently running into errors where it's trying to update EndpointSlices that are stale. This is caused by follow up calls to running before the informer cache is up to date, which happens all the time during rolling updates where rapid churn of EndpointSlices is expected. Although this eventually resolves itself, it's causing a lot more load on API Server than necessary and can potentially result in some bad states where an update fails but a delete succeeds. In this specific instance, it looks like a syncService run is occurring after the first 2 EndpointSlices have been created but before the one referencing pod1 is in the informer cache. That results in the controller creating a new EndpointSlice to reference pod1 and thus we end up with a duplicate. If anything in the future triggers another syncService call that would be resolved. I'd initially thought that EndpointSliceTracker would be thing to trigger that follow up syncService, but I was wrong. That tracker just tracks the latest resource version for each EndpointSlice the controller has created/updated and compares that with EndpointSlices received through create/update events. If the controller doesn't think that EndpointSlice should exist or the controller's resource version for that slice does not match it triggers another syncService. In this case neither of those conditions would be met so no additional syncService call would be triggered. So that's a long way of saying that I think our best possible solution here is to introduce some delay between when Services can be synced. I think the easiest solution that would solve 99% of our problems would be to add something like a 1s delay in between syncs simply by delaying when was called. This would not be ideal, but it would result in an immediate improvement and be easy to backport. It's worth noting that although duplicate EndpointSlices like this or even attempts to update a stale EndpointSlice are not ideal, they are not actually invalid and will eventually resolve themselves. Consumers of this API already have to do some form of deduping.

With that said, I'm hopeful that there's a clean way to implement a solution that actually waited for each EndpointSlice change the controller has made to be reflected in the cache because that would be a better solution. I'll spend some time looking into possible implementations here.
I've created a draft PR that contains my initial progress towards the approach I described above. I don't particularly love this approach so I'm very open to other ways we could solve this issue. It's also still very much a WIP and has significant work remaining. I opened the PR early just so I could get some initial feedback on the direction and potentially inspire other ideas around approaches we could take here.
is still the best approach I can come up with here. It is ready for review if anyone has time to take a look.
Quick update here, I've limited the scope here to the following changes as discussed in Update the flaky test to consider duplicate EndpointSlices valid. Update the docs to make it very clear that consumers of the EndpointSlice API will have to do deduplication (as kube-proxy already does). Update the controller error handling to return immediately if a create, update, or delete errors (in that order). Reviews would be appreciated on all of the above.
Reopening since is not yet merged.
Thanks for catching that Closing now that is merged. /close
Closing this issue. <detailsIn response to : Thanks for catching that Closing now that is merged. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: nc.eventRecorder.Event(namespace, api.EventTypeNormal, "DeleteNamespace", fmt.Sprintf("Marking for deletion")) _, err := nc.federatedApiClient.Core().Namespaces().Update(updatedNamespace) if err != nil { <del> glog.Errorf("Failed to update namespace %s: %v", updatedNamespace.Name, err) nc.deliverNamespace(namespace.Name, 0, true) return </del> <ins> return fmt.Errorf("failed to update namespace: %v", err) </ins> } } <del> // TODO: delete all namespace content. </del> <ins> // Right now there is just 5 types of objects: ReplicaSet, Secret, Ingress, Events and Service. // Temporarly these items are simply deleted one by one to squeeze this code into 1.4. // TODO: Make it generic (like in the regular namespace controller) and parallel. err := nc.federatedApiClient.Core().Services(namespace.Name).DeleteCollection(&api.DeleteOptions{}, api.ListOptions{}) if err != nil { return fmt.Errorf("failed to delete service list: %v", err) } err = nc.federatedApiClient.Extensions().ReplicaSets(namespace.Name).DeleteCollection(&api.DeleteOptions{}, api.ListOptions{}) if err != nil { return fmt.Errorf("failed to delete replicaset list from namespace: %v", err) } err = nc.federatedApiClient.Core().Secrets(namespace.Name).DeleteCollection(&api.DeleteOptions{}, api.ListOptions{}) if err != nil { return fmt.Errorf("failed to delete secret list from namespace: %v", err) } err = nc.federatedApiClient.Extensions().Ingresses(namespace.Name).DeleteCollection(&api.DeleteOptions{}, api.ListOptions{}) if err != nil { return fmt.Errorf("failed to delete ingresses list from namespace: %v", err) } err = nc.federatedApiClient.Core().Events(namespace.Name).DeleteCollection(&api.DeleteOptions{}, api.ListOptions{}) if err != nil { return fmt.Errorf("failed to delete events list from namespace: %v", err) } </ins> // Remove kube_api.FinalzerKubernetes if len(updatedNamespace.Spec.Finalizers) != 0 {

In PR the specific resource types are handled explicitly. Need to delete them in more generic way, in a similar way that namespace controller does.
- switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)
did fix this?
Please close or move to next-candidate with work remaining to be done.
Yes fixed by


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package filters import ( "net/http" "strings" ) // WithHSTS is a simple HSTS implementation that wraps an http Handler. // If hstsDirectives is empty or nil, no HSTS support is installed. func WithHSTS(handler http.Handler, hstsDirectives []string) http.Handler { if len(hstsDirectives) == 0 { return handler } allDirectives := strings.Join(hstsDirectives, "; ") return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { // Chrome and Mozilla Firefox maintain an HSTS preload list // issue : golang.org/issue/26162 // Set the Strict-Transport-Security header if it is not already set if _, ok := w.Header()["Strict-Transport-Security"]; !ok { w.Header().Set("Strict-Transport-Security", allDirectives) } handler.ServeHTTP(w, req) }) } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add the "Strict-Transport-Security" header to the API server responses. Why is this needed: Nessus scanner treats this as a Medium Risk in kubernetes 1.18.10.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/remove-sig api-machinery /sig auth


Question: // attachFileDevice takes a path to a regular file and makes it available as an // attached block device. func attachFileDevice(path string, exec utilexec.Interface) (string, error) { <del> blockDevicePath, err := getLoopDevice(path, exec) </del> <ins> blockDevicePath, err := getLoopDevice(path) </ins> if err != nil && err.Error() != ErrDeviceNotFound { return "", err }

I tried to use a local volume block device in minikube. Its trying to call losetup with the -j option and its failing as the losetup in minikube does not provide this option. This may effect other distro's as well. The end result is lock block devices don't work on the platform.
/sig storage
Probably caused by this code:
Did a strace on losetup -j. looks like the same info can be had by: cat /sys/block/loop0/loop/backing_file
Just had the same problem with Ceph cluster creation in minikube, the osd-prepare doesn't start because of this:
FYI: has the same problem in minikube. Replacing the /sbin/losetup busybox link with a "real" losetup helped for now, but isn't a solution of course.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help </details


Question: t.Errorf("Didn't expect the pod to be scheduled.") } <del> if filterPlugin.numFilterCalled < tt.expectFilterNumCalled { t.Errorf("Expected the filter plugin to be called at least %v times, but got %v.", tt.expectFilterNumCalled, filterPlugin.numFilterCalled) </del> <ins> if numFilterCalled := atomic.LoadInt32(&filterPlugin.numFilterCalled); numFilterCalled < tt.expectFilterNumCalled { t.Errorf("Expected the filter plugin to be called at least %v times, but got %v.", tt.expectFilterNumCalled, numFilterCalled) </ins> } if numScoreCalled := atomic.LoadInt32(&scorePlugin.numScoreCalled); numScoreCalled < tt.expectScoreNumCalled { t.Errorf("Expected the score plugin to be called at least %v times, but got %v.", tt.expectScoreNumCalled, numScoreCalled)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: integration-master Which test(s) are flaking: TestPostFilterPlugin Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /remove-sig testing
/assign


Question: framework.ExpectEqual(foundSvc, true, "could not find service 'kubernetes' in service list in all namespaces") }) <ins> ginkgo.It("should test the lifecycle of an Endpoint", func() { ns := f.Namespace.Name testEndpointName := "testservice" ginkgo.By("creating an Endpoint") _, err := f.ClientSet.CoreV1().Endpoints(ns).Create(context.TODO(), &v1.Endpoints{ ObjectMeta: metav1.ObjectMeta{ Name:      testEndpointName, Namespace: ns, Labels: map[string]string{ "testendpoint-static": "true", }, }, Subsets: []v1.EndpointSubset{{ Addresses: []v1.EndpointAddress{{ IP: "10.0.0.24", }}, Ports: []v1.EndpointPort{{ Name:     "http", Port:     80, Protocol: v1.ProtocolTCP, }}, }}, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create Endpoint") // set up a watch for the Endpoint // this timeout was chosen as there was timeout failure from the CI endpointWatchTimeoutSeconds := int64(180) endpointWatch, err := f.ClientSet.CoreV1().Endpoints(ns).Watch(context.TODO(), metav1.ListOptions{LabelSelector: "testendpoint-static=true", TimeoutSeconds: &endpointWatchTimeoutSeconds}) framework.ExpectNoError(err, "failed to setup watch on newly created Endpoint") endpointWatchChan := endpointWatch.ResultChan() ginkgo.By("waiting for available Endpoint") for watchEvent := range endpointWatchChan { if watchEvent.Type == "ADDED" { break } } ginkgo.By("listing all Endpoints") endpointsList, err := f.ClientSet.CoreV1().Endpoints("").List(context.TODO(), metav1.ListOptions{LabelSelector: "testendpoint-static=true"}) framework.ExpectNoError(err, "failed to list Endpoints") foundEndpointService := false var foundEndpoint v1.Endpoints for _, endpoint := range endpointsList.Items { if endpoint.ObjectMeta.Name == testEndpointName && endpoint.ObjectMeta.Namespace == ns { foundEndpointService = true foundEndpoint = endpoint break } } framework.ExpectEqual(foundEndpointService, true, "unable to find Endpoint Service in list of Endpoints") ginkgo.By("updating the Endpoint") foundEndpoint.ObjectMeta.Labels["testservice"] = "first-modification" _, err = f.ClientSet.CoreV1().Endpoints(ns).Update(context.TODO(), &foundEndpoint, metav1.UpdateOptions{}) framework.ExpectNoError(err, "failed to update Endpoint with new label") ginkgo.By("fetching the Endpoint") _, err = f.ClientSet.CoreV1().Endpoints(ns).Get(context.TODO(), testEndpointName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to fetch Endpoint") framework.ExpectEqual(foundEndpoint.ObjectMeta.Labels["testservice"], "first-modification", "label not patched") endpointPatch, err := json.Marshal(map[string]interface{}{ "metadata": map[string]interface{}{ "labels": map[string]string{ "testservice": "second-modification", }, }, "subsets": []map[string]interface{}{ { "addresses": []map[string]string{ { "ip": "10.0.0.25", }, }, "ports": []map[string]interface{}{ { "name": "http-test", "port": int32(8080), }, }, }, }, }) framework.ExpectNoError(err, "failed to marshal JSON for WatchEvent patch") ginkgo.By("patching the Endpoint") _, err = f.ClientSet.CoreV1().Endpoints(ns).Patch(context.TODO(), testEndpointName, types.StrategicMergePatchType, []byte(endpointPatch), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch Endpoint") ginkgo.By("fetching the Endpoint") endpoint, err := f.ClientSet.CoreV1().Endpoints(ns).Get(context.TODO(), testEndpointName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to fetch Endpoint") framework.ExpectEqual(endpoint.ObjectMeta.Labels["testservice"], "second-modification", "failed to patch Endpoint with Label") endpointSubsetOne := endpoint.Subsets[0] endpointSubsetOneAddresses := endpointSubsetOne.Addresses[0] endpointSubsetOnePorts := endpointSubsetOne.Ports[0] framework.ExpectEqual(endpointSubsetOneAddresses.IP, "10.0.0.25", "failed to patch Endpoint") framework.ExpectEqual(endpointSubsetOnePorts.Name, "http-test", "failed to patch Endpoint") framework.ExpectEqual(endpointSubsetOnePorts.Port, int32(8080), "failed to patch Endpoint") ginkgo.By("deleting the Endpoint by Collection") err = f.ClientSet.CoreV1().Endpoints(ns).DeleteCollection(context.TODO(), metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: "testendpoint-static=true"}) framework.ExpectNoError(err, "failed to delete Endpoint by Collection") ginkgo.By("waiting for Endpoint deletion") for watchEvent := range endpointWatchChan { if watchEvent.Type == "DELETED" { break } } }) </ins> }) // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed.

According to this APIsnoop query, there are still some remaining Endpoints endpoints which are untested. Create a Service in a namespace List all Endpoints in all Namespaces find the created Service(1) Replace/update the Namespaced Endpoint for the Service(1) Patch the Namespaced Endpoint for the Service(1) Get the Namespaced Endpoint for the Service(1) Delete Namespaced Endpoint(1) via a Collection with a LabelSelector Discover useragents: If a test with these calls gets merged, Conformance coverage will go up by 4 points /sig testing /sig architecture /area conformance
When you create a selector, race conditions can occur between controllers and our test code. We have services that test endpoint code, we could do these as explicit tests that are NOT handled by the endpoints controller. Let's try this with are managed by the endpoints controller w/ a selector. Endpoints normally go along with services: we want to create a service sans selector. This keeps it out of the endpoints controller, so we can test directly. Agreed for backlog. Double check that we don't already have test for that, whether it's being done by the endpoints controller that doesn't hit the API. Look at ALL the user agents that hit the endpoint.
/reopen /open Needs promotion!
Reopened this issue. <detailsIn response to : /reopen /open Needs promotion! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/retitle Promote CoreV1Endpoints test - +4 coverage
This doesn't have an APISnoop PR because (the exact process wasn't in place properly when the original mock test ticket was made)
/retitle Write CoreV1Endpoints test+promote - +4 coverage
please close this Issue in favour of and place in In Progress Issues on org dashboard
/close Close this Issue in favour of and place in In Progress Issues on org dashboard
Closing this issue. <detailsIn response to : /close Close this Issue in favour of and place in In Progress Issues on org dashboard Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close Close this Issue in favour of and place in In Progress Issues on org dashboard


Question: tar xf elasticsearch-1.5.2.tar.gz &&  rm elasticsearch-1.5.2.tar.gz <ins> RUN mkdir -p /elasticsearch-1.5.2/config/templates </ins> COPY elasticsearch.yml /elasticsearch-1.5.2/config/elasticsearch.yml <ins> COPY template-k8s-logstash.json /elasticsearch-1.5.2/config/templates/template-k8s-logstash.json </ins> COPY run.sh / COPY elasticsearch_logging_discovery /

When i doing an elasticsearch aggs with the following statement: then, i got this: Actually, i want the pod names like this:
I have solved this problem by adding a index template for k8s logs. Now, I can aggregation pod names of the same namespace.
How are you sending your logs to elasticsearch? Using the default fluentd configuration?
First, I use the default fluentd configuration. Secondly, My application is written with nodejs language, using library.


Question: klog.V(2).InfoS("Volume reconstruction finished") } <del> func (rc *reconciler) updateStatesNew(reconstructedVolumes map[v1.UniqueVolumeName]*globalVolumeInfo) { </del> <ins> func (rc *reconciler) updateStates(reconstructedVolumes map[v1.UniqueVolumeName]*globalVolumeInfo) { </ins> for _, gvl := range reconstructedVolumes { err := rc.actualStateOfWorld.AddAttachUncertainReconstructedVolume( //TODO: the devicePath might not be correct for some volume plugins: see issue #54108

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We have even encountered such case that pod continuously reports “Unable to attach or mount volumes for pod;...” when kubelet restarted. The reasons are listed in details as below: , as result of apiserver instantaneous unstablility, dswp get pvc‘s pv spec failed, then the pvc volume does not be to dsw. , syncStates() lists all the volume directories locally, and recontructs the volume from local volume, if the volume isn't in dsw, it will put the recontructed volume to asw, but now the volume.OuterVolumeSpecName isn't correct. chance, at this time, In , the fail to pvc volume is to dsw again normally in reconcile() -mountAttachVolumes(), it compares dsw and asw by volume's innerVolumename, it's all the same. but it could not sense the OuterVolumeSpecName dismatch. so it will not re-mount again to update asw cache , it checks pod's volume attach or mount success by comparing volume.OuterVolumeSpecName and , but this pv'c OuterVolumeSpecName isn't correct in asw. so pod continuously reports “Unable to attach or mount volumes for pod;...” reconcile() could sense the reconstruct volume's dismatch to the dsw, then re-mount again to make volume cache consistency Kubernetes version (use ): 1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
CC
I think this is a point that can be optimized. /cc
I will give a fix for this
Thanks for reporting the issue, this innerSpecName and outerSpecName are confusing and cause issues. Please ping me for review when you have a fix. Thanks!
Please don't triage your own issues immediately, give someone else a chance to review them :( /sig storage
okay, I know


Question: generic.ObjectCountQuotaResourceNameFor(schema.GroupResource{Resource: "pods"}): resource.MustParse("1"), }, }, <ins> "count pod overhead as usage": { pod: &api.Pod{ Spec: api.PodSpec{ Overhead: api.ResourceList{ api.ResourceCPU: resource.MustParse("1"), }, Containers: []api.Container{ { Resources: api.ResourceRequirements{ Requests: api.ResourceList{ api.ResourceCPU: resource.MustParse("1"), }, Limits: api.ResourceList{ api.ResourceCPU: resource.MustParse("2"), }, }, }, }, }, }, usage: corev1.ResourceList{ corev1.ResourceRequestsCPU: resource.MustParse("2"), corev1.ResourceLimitsCPU:   resource.MustParse("3"), corev1.ResourcePods:        resource.MustParse("1"), corev1.ResourceCPU:         resource.MustParse("2"), generic.ObjectCountQuotaResourceNameFor(schema.GroupResource{Resource: "pods"}): resource.MustParse("1"), }, podOverheadEnabled: true, }, "do not count pod overhead as usage with pod overhead disabled": { pod: &api.Pod{ Spec: api.PodSpec{ Overhead: api.ResourceList{ api.ResourceCPU: resource.MustParse("1"), }, Containers: []api.Container{ { Resources: api.ResourceRequirements{ Requests: api.ResourceList{ api.ResourceCPU: resource.MustParse("1"), }, Limits: api.ResourceList{ api.ResourceCPU: resource.MustParse("2"), }, }, }, }, }, }, usage: corev1.ResourceList{ corev1.ResourceRequestsCPU: resource.MustParse("1"), corev1.ResourceLimitsCPU:   resource.MustParse("2"), corev1.ResourcePods:        resource.MustParse("1"), corev1.ResourceCPU:         resource.MustParse("1"), generic.ObjectCountQuotaResourceNameFor(schema.GroupResource{Resource: "pods"}): resource.MustParse("1"), }, podOverheadEnabled: false, }, </ins> } for testName, testCase := range testCases { t.Run(testName, func(t *testing.T) { <ins> defer featuregatetesting.SetFeatureGateDuringTest(t, feature.DefaultFeatureGate, features.PodOverhead, testCase.podOverheadEnabled)() </ins> actual, err := evaluator.Usage(testCase.pod) if err != nil { t.Error(err)

assume that: cluster with only one node with 4G allocatable memory Define test runtimeclass with PodOverhead Define resourceQuota test under test namespace with 4Gi memory limit Create Pod with test runtimeClass under test namespace podOverhead resource not counted into resourceQuota used info podOverhead resource counted into node requests info try to create one more pod with 2G memory requests, resourceQuota check passed but failed to schedule because it requests 2G+200M memory podOverhead resource should be counted into resourceQuota used info to keep resourceQuota check and schedule check be consistent. Kubernetes version 1.16
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node
ref:
-- can you verify that you have the feature gate enabled for pod overhead for api-server, as well as kubelet and scheduler? Example:
I made a cluster with the following command, using crio as a container runtime( is not available with Docker) And the issue was reproduced.
overhead feature enabled, the overhead resource has been counted into requests during scheduling but not counted into resourcequota
Currently, PodOverhead is defined as a fixed resource. this value looks like the average usage value and being into requests resource during scheduling. type Overhead struct { PodFixed *ResourceList } My use case is kata_container, the VMM related processes consume much overhead resource when pod running with high workload. A small overhead(small cgroup limits) will trigger oom but a big value result that big overhead requests take into account during scheduling. So how about define PodOverhead including limits and requests like that in Pod resources definition. It could be helpful resources: limits: memory: 1Gi requests: memory: 200M
CC: FYI


Question: GCE_SOAK_CONTINUOUS_SKIP_TESTS=( "Density.*30spods" "Elasticsearch" <del> "Etcd.*SIGKILL" </del> "externalsloadsbalancer" "identicallysnamedsservices" "networkspartition" "Services.*Typesgoessfrom" <ins> "${DISRUPTIVE_TESTS[@]}"       # avoid component restarts. </ins> ) GCE_RELEASE_SKIP_TESTS=(

We should be testing the reliability of a long-running cluster. These tests kill/restart components or nodes in the cluster, defeating the purpose of a soak cluster.
Example tests: NodessResize Services.restarting DaemonRestart NodessResize Reboot Etcd failure
Agreed. Some of them already seem to be skipped: GCESOAKCONTINUOUSSKIPTESTS=( "Density.30spods" "Elasticsearch" "Etcd.SIGKILL" "externalsloadsbalancer" "identicallysnamedsservices" "networkspartition" "Services.Typesgoessfrom" ) Q
FYI
It'd be hard to prevent regression as more tests are . It might be better if we explicitly define a set of DISRUPTIVETESTS, and include them wherever needed, e.g., GCEPARALLELSKIPTESTS and GCESOAKCONTINUOUSSKIPTESTS. That way, people who are adding tests will be aware of the category. WDYT?
Agreed, that seems like a good idea. Want to send in a PR? Or at least open an issue?


Question: go_test( name = "go_default_test", srcs = [ <ins> "init_windows_test.go", </ins> "server_bootstrap_test.go", "server_test.go", ],

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Windows nodes running with high (90%) CPU usage are having calls to /stats/summary time out with "Context Deadline Exceeded". When this happens metrics-server cannot gather accurate CPU/memory usage for the node or pods running on the node which results in HPA not functioning returning <unknowninstead of values What you expected to happen: Windows nodes should be able to run with high CPU utilization without negatively impacting Kubernetes functionality How to reproduce it (as minimally and precisely as possible): a cluster with Windows Server 2019 nodes CPU intensive workloads into cluster ex: workload to put node into high CPU state node metrics with Anything else we need to know?: This issue appears to be easier to reproduce on 1.18.x nodes vs 1.16.x or 1.17.x nodes Ex: ! ! metrics-server logs show E1020 10:58:24. 1 ] unable to fully collect metrics: unable to fully scrape metrics from source kubeletsummary:akswin000000: unable to fetch metrics from Kubelet akswin000000 (10.240.0.115): Get https://10.240.0.115:/stats/summary?onlycpuandmemory=true: context deadline exceeded Kubelet logs show and Environment: Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: AKS / Azure OS (e.g: ): WIndows Server 2019 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: Docker EE version 19.3.11
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig windows /area kubelet
I was able to reproduce on 1.18.8 during a scale operation: where it fully subscribes all CPU's in Allocatable: Metric server logs: I do not see "context deadline" in kubelet but I did find some logs assocated: Once the pods are scheduled the node remains at 100% cpu and I no longer see errors but is not accurate: on the node (run after the pod deployment stabilized)
Adding to kubelete's flag and I no longer see the errors. This reserves some cpu for kubelet and other processes. We call this out in the but do not do the same for cpu.
Also, without setting system reserved CPU, it looks like does not accurately reflect CPU usage on the nodes when CPU utilization is very high. When I got some nodes into this state I did not see any error messages in logs from metrics-server and also I noticed that that querying the metrics API directly from showed recent timestamps.
I run into this again during scaling, collectiong logs for analysis here: Metric Server logs: Kubelet logs:
have you experienced this at all?
I believe has seen high CPU issues before. Not particularly on this. I've heard there were kernel issues in the past that triggered high CPU cases.
Some investigations on our end are showing that the DiagTrack service running in the infra/pause container can consume non-trivial amounts of CPU. I'll create an issue to address that problem as it might help out some here. That certainly will not eliminate the problem we are seeing tho.
is an issue for DiagTrack
The differences in behavior between 1.17. and 1.18. clusters looks like it is caused by (which was NOT backported to any prior releases) K8s version <= 1.17 behavior Prior to these changes CPU weights for containers were inferred based on limit values set in the pod specs. The observed behavior on Windows system when CPU weights were used was that under high-CPU load pods/containers were not guaranteed their scheduling time. The good news here is that system critical services like kubelet/docker/etc were still able to get enough CPU time for the nodes to function properly. K8s version = 1.18 behavior After these changes CPU limits were enforced for pods.

The observed behavior on 1.18 clusters is the when a Windows node enters a high-CPU state is that containers are guaranteed their scheduling time but host resources start to get starved. Usually this manifests as the node entering the state, metrics not working correctly (because calls to /stats/summary are timing out) or both. Suggestion / possible fixes Set the Windows process priority of the kubelet to  to help ensure it gets CPU time Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) Set reserve CPU with with --system-reserves=[cpu=500m] This needs to scaled based on number of CPUs on host based on my experimentations Currently guidance on for Windows only mentions reserving system-memory It is still possible to over-commit CPU and starve kubelet/docker/HCS if users schedule pods with mis-matching requests and limits Update documentation.
Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) I can work on this, I am wondering if we should do the same for CRI(like docker) and CNI processes.
Sure, if you want to work on this go ahead. I think we should probably try to make this be toggleable via a kubelet flag. I believe there are other windows specific flags already implemented in the kubelet (like one to run as a windows service IIRC)
docker/containerd should probably have a flag that allows them to run as high priory as well but would likely be different issues in the respective projects
/reopen I don't think this was ever fully addressed
Reopened this issue. <detailsIn response to : /reopen I don't think this was ever fully addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
also related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: NumberAvailable:        0, }, <del> msg:  "Waiting for rollout to finish: 0 out of 1 new pods have been updated...n", </del> <ins> msg:  "Waiting for daemon set "foo" rollout to finish: 0 out of 1 new pods have been updated...n", </ins> done: false, }, {

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In our CI, we run multiple times in parallel when we update multiple deployments. This lets us see the status of all deployments as they run. When a deployment succeeds, the final message tells us the name of the deployment. However, the intermediate "Waiting for rollout to finish" messages do not say the name of the deployment. This means that in order to look at multiple rollouts in parallel, we need to manually pipe through another command to prefix it with the rollout name. What you expected to happen: Each line to mention the rollout name. How to reproduce it (as minimally and precisely as possible): Update any deployment, run , note that intermediate status names don't contain the deployment name. Anything else we need to know?: Arguably another fix would be to allow you to pass multiple rollouts to and have kubectl handle polling multiple deployments in parallel. Environment: This is still true on master by code inspection.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) } } <ins> func TestReuseRequest(t *testing.T) { var tests = []struct { name        string enableHTTP2 bool }{ {"HTTP1", false}, {"HTTP2", true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte(r.RemoteAddr)) })) ts.EnableHTTP2 = tt.enableHTTP2 ts.StartTLS() defer ts.Close() ctx, cancel := context.WithCancel(context.Background()) defer cancel() c := testRESTClient(t, ts) req1, err := c.Verb("GET"). Prefix("foo"). DoRaw(ctx) if err != nil { t.Fatalf("Unexpected error: %v", err) } req2, err := c.Verb("GET"). Prefix("foo"). DoRaw(ctx) if err != nil { t.Fatalf("Unexpected error: %v", err) } if string(req1) != string(req2) { t.Fatalf("Expected %v to be equal to %v", string(req1), string(req2)) } }) } } func TestHTTP1DoNotReuseRequestAfterTimeout(t *testing.T) { var tests = []struct { name        string enableHTTP2 bool }{ {"HTTP1", false}, {"HTTP2", true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { done := make(chan struct{}) ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { t.Logf("TEST Connected from %v on %vn", r.RemoteAddr, r.URL.Path) if r.URL.Path == "/hang" { t.Logf("TEST hanging %vn", r.RemoteAddr) <-done } w.Write([]byte(r.RemoteAddr)) })) ts.EnableHTTP2 = tt.enableHTTP2 ts.StartTLS() defer ts.Close() // close hanging connection before shutting down the http server defer close(done) ctx, cancel := context.WithCancel(context.Background()) defer cancel() transport, ok := ts.Client().Transport.(*http.Transport) if !ok { t.Fatalf("failed to assert *http.Transport") } config := &Config{ Host:      ts.URL, Transport: utilnet.SetTransportDefaults(transport), Timeout:   100 * time.Millisecond, // These fields are required to create a REST client. ContentConfig: ContentConfig{ GroupVersion:         &schema.GroupVersion{}, NegotiatedSerializer: &serializer.CodecFactory{}, }, } if !tt.enableHTTP2 { config.TLSClientConfig.NextProtos = []string{"http/1.1"} } c, err := RESTClientFor(config) if err != nil { t.Fatalf("failed to create REST client: %v", err) } req1, err := c.Verb("GET"). Prefix("foo"). DoRaw(ctx) if err != nil { t.Fatalf("Unexpected error: %v", err) } _, err = c.Verb("GET"). Prefix("/hang"). DoRaw(ctx) if err == nil { t.Fatalf("Expected error") } req2, err := c.Verb("GET"). Prefix("foo"). DoRaw(ctx) if err != nil { t.Fatalf("Unexpected error: %v", err) } // http1 doesn't reuse the connection after it times if tt.enableHTTP2 != (string(req1) == string(req2)) { if tt.enableHTTP2 { t.Fatalf("Expected %v to be the same as %v", string(req1), string(req2)) } else { t.Fatalf("Expected %v to be different to %v", string(req1), string(req2)) } } }) } } func TestTransportConcurrency(t *testing.T) { const numReqs = 10 var tests = []struct { name        string enableHTTP2 bool }{ {"HTTP1", false}, {"HTTP2", true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { t.Logf("Connected from %v %v", r.RemoteAddr, r.URL) fmt.Fprintf(w, "%v", r.FormValue("echo")) })) ts.EnableHTTP2 = tt.enableHTTP2 ts.StartTLS() defer ts.Close() var wg sync.WaitGroup wg.Add(numReqs) c := testRESTClient(t, ts) reqs := make(chan string) defer close(reqs) for i := 0; i < 4; i++ { go func() { for req := range reqs { res, err := c.Get().Param("echo", req).DoRaw(context.Background()) if err != nil { t.Errorf("error on req %s: %v", req, err) wg.Done() continue } if string(res) != req { t.Errorf("body of req %s = %q; want %q", req, res, req) } wg.Done() } }() } for i := 0; i < numReqs; i++ { reqs <- fmt.Sprintf("request-%d", i) } wg.Wait() }) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !

transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.


Question: apiVersion: apps/v1 kind: DaemonSet metadata: <del> name: npd-v0.8.7 </del> <ins> name: npd-v0.8.8 </ins> namespace: kube-system labels: k8s-app: node-problem-detector <del> version: v0.8.7 </del> <ins> version: v0.8.8 </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec: selector: matchLabels: k8s-app: node-problem-detector <del> version: v0.8.7 </del> <ins> version: v0.8.8 </ins> template: metadata: labels: k8s-app: node-problem-detector <del> version: v0.8.7 </del> <ins> version: v0.8.8 </ins> kubernetes.io/cluster-service: "true" spec: containers: - name: node-problem-detector <del> image: k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.7 </del> <ins> image: k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.8 </ins> command: - "/bin/sh" - "-c"

/cc Can and npd v0.8.8(with ) fix this problem? /sig node Example: node-problem-  /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
Yes, the and npd v0.8.8(with kubernetes/node-problem-detector) should fix this problem
/triage accepted /assign
There are other files that are needed to be updated, for reference Could you please update.


Question: #run the script once to download the dependent java libraries into the image RUN mkdir /output <del> RUN build/gen-swagger-docs.sh v1 https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes/master/api/swagger-spec/v1.json https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes/master/pkg/api/v1/register.go </del> <ins> RUN mkdir /swagger-source RUN wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/swagger-spec/v1.json -O /swagger-source/v1.json RUN build/gen-swagger-docs.sh v1 https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes/master/pkg/api/v1/register.go RUN rm /output/* RUN rm /swagger-source/* </ins> ENTRYPOINT ["build/gen-swagger-docs.sh"]

From gen-swagger-docs should read the local swagger spec rather than fetching it from That way it generates release api-reference when run-gen-swagger-docs is run in release branch and it generates api-reference for HEAD code, when the script is run on HEAD.
Assigning to as per discussion.
Blocks


Question: found := true writer := newLogWriter(stdout, stderr, opts) msg := &logMessage{} <ins> baseName := filepath.Base(path) dir := filepath.Dir(path) </ins> for { if stop || (limitedMode && limitedNum == 0) { klog.V(2).InfoS("Finished parsing log file", "path", path)

won't output logs after kubelet rotate log filer of the container. kubectl output logs correctly after the log file is rotated. the config to . So kubelet will rotate the log soon. Build a image with this program a deploy with the previous image. the command No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/area kubelet
/triage accepted /priority important-longterm
/assign since you sent a PR already
Ran into this issue!


Question: klog.V(5).Infof("Delete service (%s): START clusterName=%q", serviceName, clusterName) serviceIPToCleanup, err := az.findServiceIPAddress(ctx, clusterName, service, isInternal) <del> if err != nil { </del> <ins> if err != nil && !retry.HasStatusForbiddenOrIgnoredError(err) { </ins> return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: controller-manager is stuck in deleting an incorrectly configured LB type service. it cannot be recovered until restarted. What you expected to happen: controller-manager should recover. At least it should allow us to create the service without restarting it. How to reproduce it (as minimally and precisely as possible): put a random IP in above yaml and make sure the resource group doen't exist. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig azure
I also want to add that the above repro is stuck in 403. It's possible for it to be stuck in 404 as well when the SP has the right permission.
Can you share the log for 404?
Thanks for reporting the issue. /assign Do you mean other services couldn't be configured when this happens?
correct.
FYI, we had the same issue when the Idle Timeout annotation was set to "3" which isn't a supported case. Will this PR also cover this edge case ?
Unfortunately , this is still not fixed. The idle timeout is not 403 or 404. We need another fix for this (and possibly more for other annotations)


Question: } // Create the deployment for CoreDNS or retain it in case the CoreDNS migration has failed during upgrade <del> if hasCoreDNSMigrationFailed { </del> <ins> if !canMigrateCorefile { </ins> if err := apiclient.CreateOrRetainDeployment(client, coreDNSDeployment, kubeadmconstants.CoreDNSDeploymentName); err != nil { return err }

What happened: Coredns failed to start in 1.17 deployment when it tried to retain a backed up Corefile with error message: Caddyfile via flag: open /etc/coredns/Corefile: no such file or directory What you expected to happen: Coredns starts normally with the backed-up Corefile How to reproduce it (as minimally and precisely as possible): Deploy a k8s 1.17 cluster and trigger coredns Corefile migration and make it fail. In our case, this was triggered in a three master -setup with 'old' method of running 'kubeadm init' for each master. Also the core-dns docker image used was tagged with '1.6.7-1', apparently causing the corefile migration version check to fail Anything else we need to know?: This seems to be caused by Both 'key' and 'path' are set to 'Corefile-backup' when this function is called. Should the 'path' always stay as 'Corefile' because that is where the container expects to find the config file? Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-02-16T17:51:57Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-02-16T17:47:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Custom deployment in Openstack environment OS (e.g: ): NAME="SLES" VERSION="15-SP1" VERSIONID="15.1" PRETTYNAME="SUSE Linux Enterprise Server 15 SP1" Kernel (e.g. ): 12.14-197.29-default Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
/assign
/area kubeadm


Question: "k8s.io/kubernetes/pkg/api" apierrors "k8s.io/kubernetes/pkg/api/errors" <ins> storageerr "k8s.io/kubernetes/pkg/api/errors/storage" </ins> "k8s.io/kubernetes/pkg/api/unversioned" "k8s.io/kubernetes/pkg/fields" "k8s.io/kubernetes/pkg/labels"

Honestly I'm not sure if this is a bug or a feature, but didn't expect it. Filing this to track.
I think it's a bug. Nothing in the code base depends on it that I can see (edit: namespace deletion does, but only via an in-server call which I'd like to see fixed), and we don't let other delete-related fields mutate during update (deleteGracePeriodSeconds)


Question: REGISTRY?=staging-k8s.gcr.io IMAGE?=debian-hyperkube-base <del> TAG=0.9 </del> <ins> TAG=0.10 </ins> ARCH?=amd64 CACHEBUST?=1

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Cloud providers are using the command-line tool to retrieve status and format volume mounts. is missing from the hyperkube image resulting in missing volumes. Issues: What you expected to happen: Volume mounts on cloud providers are expected to be formatted and mounted. How to reproduce it (as minimally and precisely as possible): GCE and Openstack volume mounts are effected. Anything else we need to know?: Environment: Kubernetes version (use ): 1.9, 1.10+ Cloud provider or hardware configuration: GCE and Openstack OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/cc
/sig storage

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): container-os Kernel (e.g. ): Linux gke-wordpress-cluster-default-pool--m764 4.4.21+ SMP Fri Feb 17 15:34:45 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux Install tools: Others: What happened: ` I was able to bring up wordpress okay the first time, except GKE wasn't creating loadbalancer ips due to a quota issue, which I resolved, note at this point the mysql pod was up and had attached to it's volume. Upon deleting the wordpress deployment and creating it again I started getting the above errors. I deleted the mysql pod as well and brought it up again to see that it had the same issue. The volumes are backed by a gluster cluster on GCE. Looking at the brick logs on one of the gluster nodes I see I've tried restarting kubelet on the node. I can't find the kubelet log file on any of gke nodes. And I don't know how to get the kube-controller and apiserver logs from the master (GKE). I suspect it's a failing of the glusterfs client on the gke nodes? What you expected to happen: Deployment to mount the volumes successfully. How to reproduce it (as minimally and precisely as possible): Not sure how, but I've run into this intermittently Anything else we need to know: The volumes are backed by gluster
/cc
If you delete pod and create pods soon, it is possible that you could see some errors like "timeout expired waiting for volumes to attach/mount" temporally. Could you wait for a few more minutes to see whether the pods could be started correctly?
I actually tried that, deleted the pods, then waited 5 minutes and recreated them with the same error. Maybe it wasn't long enough
Can you post output of ? It may have some other events for failure..
I deleted the cluster this afternoon, but I checked constantly and those were the only errors, Timeout on mount and a failed to sync error
If you use level 4 log, it should also give the detailed messages of failed mount so that we can figure out why it failed.

Can you email me your project name, cluster name, and zone information? I can take a look at the master logs to see if there's anything useful there.
This is starting to happen more frequently..
could you provide more details about in what situation it happened frequently? Kubelet log could give us more information. If you use GKE cluster, run "journctlctl -u kubelet /tmp/log" to get the log. You could also share with us the log if it is ok.
This seems to be the relevant part of the log I re-created that and my pods sprang back up. Weird thing is I had created that, along with the pvcs and pvs but seems it's somehow getting deleted. There's almost certainly certain I'm doing wrong but I'm not sure what. ]
HI Jonathan, can you clarify what you had to recreate?
The glusterfs endpoints , Michelle Au <:
I have a very similar problem running OpenShift 3.4 in AWS EC2 doing the same exact thing. Brand new OSE install [root master]# oc version oc v3.4.1.12 kubernetes v1.4.0+ features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.31.40.126:8443 openshift v3.4.1.12 kubernetes v1.4.0+ POD in question is the docker-registry (kind of important for OSE to work) I've followed this to see if it would fix the issue: But no luck or change. Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] then: Unable to mount volumes for pod "docker-registry-3-afbbt_default(-1a45-11e7-abf8-)": timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] I have no idea what log(s) to look at, this was of no help when I deployed a new docker-registry: oc logs -f dc/docker-registry Any help or guidance would be greatly appreciated as this is a blocker issue for me.

thanks and best wishes,
are you using glusterfs for registry-storage volume or other type of volume? Could you share us your kubelet log if possible over slack Thanks!
Hi Jing Xu, Would you know where this log would be? Or is the name kublet<somethingso I may find it for you? I just requested to join the slack channel, as soon as I'm signed in I will share the log. Probably tomorrow as I'm on the east coast USA Thank you! Joel
For you are using GKE with GCI image, you can use "journalctl -u kubelet " to get the log. Otherwise, it should be in " , shepp67 <:
I also experienced this. Kubernetes version:Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Install tools: custom That kubernetes worker vm actually had volumes attached. I saw it on OpenStack dashboard. But: and actually: Besides, this is a new worker vm. I have other worker vms already running, using cinder volumes and there were no such errors. I use Helm to deploy this Pod. I have deployed it earlier and removed it. But it was 2 days ago. I found the solution here: . On that worker vm I run: Edit: or upgrade Kubernetes to = 1.6.0-beta.3 I don't know if this is the solution to issue.
I'm seeing this same error on all my manually launched private Ubuntu based clusters (6), running 1.5.3 and 1.6.0. In my case the messages reference only stopped pods launched by CronJob Jobs. The messages indicate issues mounting every volume in the pod including PVs and node-local volumes. These pods, and future ones launched by these CronJobs, are fully successful when they run. It is only after they run to successful completion and exit that these messages show up. Glancing at the kubelet code, this appears to be coming out of some housekeeping/synch loop.
I've identified a bug and am working on a fix for when "Unable to mount volumes for pod..." + "Error syncing pod" happen indefinitely BUT the pod is actually running just fine, as observed by .

For others, if you see this message but your pod is Running, you can safely ignore it. It's caused by a small limitation of the reconstruction of state (what volumes are mounted) so there is a chance you will see this only if you restart kubelet while your pods had volumes mounted. correction: you say they even Completed, they weren't just Running fine. So yeah, it's just misleading logging and not really an indication of anything breaking
Because the error message is very generic, I think there might be different problems mentioned in this thread. 's issue could be fixed by 's PR are you still have any issues with mount? could you also let us know whether you still have the issue? Thanks! Jing
Hi my issue was related to which helped me sort out. I was creating the gluster endpoints without a service and turns out the kube-controller deletes all such endpoints that are not attached to a service at regular intervals. So after every so often when a pod went down and tried to come up again and mount its respective pv it failed because it couldn't connect to gluster. The fix is to create a service for the gluster endpoints.
I also just ran into such timeout issue: After running a load test against my postgres backed web application, the database didn't respond anymore for still unknown reasons. According to kubernetes it was up and running The log showed a bunch of these messages. Not sure yet why but possibly it couldn't access the volume anymore: Short after that, the pod entered "error" state: The event log showed: This error disappeared after a few minutes (not sure how long exactly, but it took a while), but then I got the same error as described here: I couldn't figure out anything more on my own. Nothing in the syslog, nothing more than this in the kubelet log. Finally I rebooted the machine which fixed the issue.
cc do you know what error of the network not ready? It might be helpful if you could share the kubelet log with us to debug. Thanks!
AFAIK, the "network not ready error" is benign, kubelet will retry after this error () cc
I hope I didn't miss some secrets in there, quite a talky log. But here it is: The database didn't respond anymore because it ran OOM and was killed, as also shown in the log.

Although it appears like it was OOM killed after it became unresponsive in the log.. Either way, a bunch of other things got killed too, so granted - "not ideal" situation :). Yet it shouldn't drop the volume and should re-attach it.
I think you are hitting the issue mentioned above. He has a PR to fix it. I will make sure it can merge soon. Thanks!
Great, thanks! Do you know roughly when this will be available in GCE?
you mean GKE or GCE? For GKE it will take a little bit longer. On GCE, when the PR gets merged, the fix should be available when the next release is out (I think should be soon)
Running on GKE. I'd rather not launch my product with this bug around. Is there a workaround for it or something I can monitor for? Or maybe it could get hot fixed on GKE? If there is no workaround, this effectively renders GKE useless for stateful stuff.
It's hard to say if volumes are actually being dropped/unmounted or if the FailedMount errors are wrong & misleading, so I wouldn't go so far as to say it's broken because of some volume issues. It could be something else entirely. If nothing else, the PR jing's referring to will clean up the event noise and help us debug the root cause. In the meantime, I wouldn't trust the FailedMount events to be accurate, so if your applications are unresponsive, there may be other causes worth considering
Well, I only can say that my volumes were dropped and it didn't recover on it's own (only after reboot). Dropped a seen by kubernetes, dunno if the volumes were still attached but they weren't usable by my scheduled database. To be clear: I'm not "complaining" and this was just test/R&D, but it would this is the scariest kind of issue for me to happen in prod, given that it's a managed service where I can't find a workaround on my own. Rebooting fixed it in my case, but that's a big hammer..
You don't need to reboot the node. Delete the pod and restart it should work for this issue. Is this workaround ok for your case? , discordianfish <:
Good to know, thanks!

Will try this next time.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hi, I'm facing the same issue with k8s 1.9, installed by kops on AWS. volume created successfully on AWS and attached to the right node, but the pod is failed to start.
/remove-lifecycle stale
could you please provide more details about your issue? What are the steps to reproduce it? Thanks!
yeah can you open a new bug for EBS and cc me with details? I was hoping we fixed most of EBS problems in 1.9 - I will be more than happy to walk through and find what is causing volume to not mount. As I understand this is different from typical EBS problems where volume fails to attach. In your case, volume is being attached to right node but it is not being seen by kubelet.
I found the issue, and it was the fact that i used c5 instances with nvme, this is why the volume never been attached as k8s looking for /dev/xvdXX and with nvme that path is different. I did notice that in 1.9 k8s should support nvme natively but it seems that it's still not stable. thanks
oh weird. You should open a bug about NVMe support then. I was thinking it would just work.
cc Is there an issue related to nvme support? please let us know any issues you still have with 1.9. Thanks!
This is nvme on AWS, not related to the work is doing on gce.
Thanks Definitely related to NVMe support. Ran into this trying to mount a gp2 EBS volume on a node which was an m5 EC2 instance.
I think I am seeing like this with , but unsure of "why" at the moment. It would seem works perfectly fine.
did you guys had a chance to open a BUG for NVMe support on EBS? If you do - please paste output of and PersistentVolume spec.

We can't fix it unless we know bit more. :(
sorry, we worked around this by reverting back to m4 class instances. It should be simple enough to reproduce however... create a k8s cluster w/ m5 EBS backed nodes and create a simple pod w/ a persistent volume. You'll notice that the pod will never launch and you should be able to debug further. If I do run into this again I'll be sure to get some more data for you.
I too have the error, error syncing pod. kubectl describe pod kafka1-kafka-0 .... .... .... Warning FailedSync 5m (x141 over 35m) kubelet, 10.186.. Error syncing pod Warning BackOff 57s (x164 over 35m) kubelet, 10.186.. Back-off restarting failed container kubectl logs kafka1-kafka-0 -p export KAFKABROKERID=kafka1-kafka-0 export KAFKAADVERTISEDLISTENERS=PLAINTEXT://172.30.*.:9092 /etc/confluent/docker/run sh: 3: /etc/confluent/docker/run: not found


Question: t.checkDNSRecordFrom( fmt.Sprintf("%s.%s.svc.%s", serviceNameLocal, f.Namespace.Name, framework.TestContext.ClusterDNSDomain), func(actual []string) bool { <del> return len(actual) >= 1 && actual[0] == fooHostname+"." && actual[1] == "2001:db8::29" </del> <ins> return len(actual) >= 2 && actual[0] == fooHostname+"." && actual[1] == "2001:db8::29" </ins> }, "cluster-dns-ipv6", moreForeverTestTimeout)

We've seen this line of panic from time to time. It is caused by an index out of bounds. The array length check on that line seems to be incorrect. We see this flakiness when running in ipv6 or dualstack-ipv6-primary environments. We do not see this flakiness in ipv4 environments. We've seen these flakes over several months. No response incorrect array length check on , and the right most is out of bounds. No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/milestone 1.25 as this is a networking conformance test that fails
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.25 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.25
/sig network
/assign
thanks for catching it
There is confusion whether or not this is a conformance test. We don't believe it is. The test description does not include . I'll remove the label from the PR.


Question: for i, uc := range uss.clients { uss.integrators[i] = fq.NewIntegrator(uss.clk) fsName := fmt.Sprintf("client%d", i) <del> uss.expectedInqueue = uss.expectedInqueue + fmt.Sprintf(`				apiserver_flowcontrol_current_inqueue_requests{flowSchema=%q,priorityLevel=%q} 0%s`, fsName, uss.name, "n") </del> <ins> uss.expectedInqueue = uss.expectedInqueue + fmt.Sprintf(`				apiserver_flowcontrol_current_inqueue_requests{flow_schema=%q,priority_level=%q} 0%s`, fsName, uss.name, "n") </ins> for j := 0; j < uc.nThreads; j++ { ust := uniformScenarioThread{ uss:    uss,

<!-- Please only use this template for submitting enhancement requests --What would you like to be : API priority and fairness metrics have labels such as "priorityLevel" and "flowSchema". These should be "prioritylevel" and "flowschema" respectively. These metrics have been to an exception list: Why is this needed: snake_case is consistent with other metrics. /sig api-machinery /kind cleanup /cc Do we want to do this before 1.20 beta?
yes, lets do that.
/triage accepted Yes, we do not want to go to beta with known wrongness in the metrics.


Question: return fmt.Errorf("one or more files must be specified as -f rsrc.yaml or --filename=rsrc.json") } } <del> if len(o.newAnnotations) < 1 && len(o.removeAnnotations) < 1 { </del> <ins> if len(o.newAnnotations) < 1 && len(o.removeAnnotations) < 1 && !o.list { </ins> return fmt.Errorf("at least one annotation update is required") } return validateAnnotations(o.removeAnnotations, o.newAnnotations)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : add --list option to kubectl annotate command Why is this needed: like kubectl label commandshow annotations quickly
/sig cli These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/assign


Question: *flexVolume // Runner used to teardown the volume. runner exec.Interface <del> volume.MetricsNil </del> } var _ volume.Unmounter = &flexVolumeUnmounter{}

kind feature What happened: Currently there is no mechanism to report metrics as the flex plugins are not enabled to collect metrics. All flex volumes lack this feature as compared to in-tree/built-in drivers even though it is possible to collect such metrics for most of the plugins. What you expected to happen: Depending on the availability of the metrics from the mounted FS, flex plugins should allow to report metrics for the FS. How to reproduce it (as minimally and precisely as possible): The default behavior that can be proved easily by just running a command similar to the following command on a node that is running a flex volume. This will show that there are no metrics collected for flex volumes. kubectl proxy $curl 127.0.0.1:8001/api/v1/nodes/<IP/proxy/stats/summary Environment**: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: local x86 OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux KubeStorage 4.4.0-130-generic -Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Others: /sig storage
+1 Yes, it will be good, if flex volume can support atleast the default metrics( i.e ), which most of the volume plugins support now. Based upon flex plugin "capabilities" parameters, it can be decided, whether the plugin supports "default metrics" or "nil metrics".


Question: servicehelpers "k8s.io/cloud-provider/service/helpers" "k8s.io/klog/v2" azcache "k8s.io/legacy-cloud-providers/azure/cache" <ins> "k8s.io/legacy-cloud-providers/azure/retry" </ins> utilnet "k8s.io/utils/net" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: controller-manager is stuck in deleting an incorrectly configured LB type service. it cannot be recovered until restarted. What you expected to happen: controller-manager should recover. At least it should allow us to create the service without restarting it. How to reproduce it (as minimally and precisely as possible): put a random IP in above yaml and make sure the resource group doen't exist. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig azure
I also want to add that the above repro is stuck in 403. It's possible for it to be stuck in 404 as well when the SP has the right permission.
Can you share the log for 404?
Thanks for reporting the issue. /assign Do you mean other services couldn't be configured when this happens?
correct.
FYI, we had the same issue when the Idle Timeout annotation was set to "3" which isn't a supported case. Will this PR also cover this edge case ?
Unfortunately , this is still not fixed. The idle timeout is not 403 or 404. We need another fix for this (and possibly more for other annotations)


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> # Based heavily on # https://github.com/dylanmei/docker-zeppelin/blob/master/Dockerfile # (which is similar to many others out there), but rebased onto maven # image. # # This image is a composition of the official docker-maven # Docker image from https://github.com/carlossg/docker-maven/ and # spark-base. </del> <ins> # This image relies on the zeppelin-build image to get the zeppelin # binaries built, and the Makefile to put it in this directory. </ins> FROM gcr.io/google_containers/spark-base:latest <del> ENV ZEPPELIN_TAG  v0.5.5 ENV MAVEN_VERSION 3.3.3 ENV SPARK_MINOR   1.5 ENV SPARK_PATCH   1 ENV SPARK_VER     ${SPARK_MINOR}.${SPARK_PATCH} ENV HADOOP_MINOR  2.6 ENV HADOOP_PATCH  1 ENV HADOOP_VER    ${HADOOP_MINOR}.${HADOOP_PATCH} RUN curl -fsSL http://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz | tar xzf - -C /usr/share  && mv /usr/share/apache-maven-${MAVEN_VERSION} /usr/share/maven  && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn ENV MAVEN_HOME /usr/share/maven # libfontconfig is a workaround for # https://github.com/karma-runner/karma/issues/1270, which caused a # build break similar to # https://www.mail-archive.com/users@zeppelin.incubator.apache.org/msg01586.html RUN apt-get update  && apt-get install -y net-tools build-essential git wget unzip python python-setuptools python-dev python-numpy libfontconfig  && apt-get clean  && rm -rf /var/lib/apt/lists/* RUN git clone https://github.com/apache/incubator-zeppelin.git --branch ${ZEPPELIN_TAG} /opt/zeppelin RUN cd /opt/zeppelin &&  mvn clean package  -Pspark-${SPARK_MINOR} -Dspark.version=${SPARK_VER}  -Phadoop-${HADOOP_MINOR} -Dhadoop.version=${HADOOP_VER}  -Ppyspark  -DskipTests &&  rm -rf /root/.m2 &&  rm -rf /root/.npm &&  echo "Successfully built Zeppelin" </del> <ins> # Expands to /opt/zeppelin/ ADD zeppelin.tgz /opt/ </ins> ADD zeppelin-log4j.properties /opt/zeppelin/conf/log4j.properties ADD zeppelin-env.sh /opt/zeppelin/conf/zeppelin-env.sh ADD docker-zeppelin.sh /opt/zeppelin/bin/docker-zeppelin.sh

The image by is over a gig. Investigate how to trim this down. (Cut out interpretters? Make them more modular somehow, possibly composing them once we have some solution like ?) It's not even clear there's upstream support for making this more modular, though.
cc
remove mvn tooling and all the extra steps, once upstream releases 's like the rest of apache.
I can look at that, but I think I can actually just strip the image of the dev tooling right now. Right now I think the Zeppelin deps themselves are a ridiculous portion of this, though. (I'll poke at it.)
(3) is an install of [bzr git mercurial openssh-client subversion] in the base java image, which seems wasteful to me. you'll have to switch base images to get rid of this. (2) is your install of [curl net-tools build-essential git wget unzip python python-setuptools python-dev python-numpy libfontconfig]. i expect curl is at least redundant, as is git if your using the java image. you should merge this layer w/ (1) and do an uninstall of [python-setuptools python-dev etc] & deps at the end. of course, those are less significant than (1), which is the mvn build. the long term solution here is to use the upstream binaries, when they exist. in the meantime, you should clone zeppelin into tmp, copy out the resulting build artifact and delete the clone. that should at least cut the size of this layer in half, as all the intermediate build artifacts are likely still present and in the final tarball/jar.
I can trim it down some by cutting out the build machinery (and I'll put up a PR shortly), but here's what the looks like: And expanded: (Instead of just deleting, it's actually easy to build, copy out, compose, too.)
Is it me or are all the containers rather large. For example, does spark itself need to be a gig? 1.5.1v3 6 weeks ago 989.7 MB There's some spark binary package that's 73M uncompressed:
Java runtime + baseimage + transitive closer is what is causing most of the bloat. But as we have discussed on other tickets, these images need to move out of the main repository.

I'm afraid I don't understand "transitive closer". For example I see my JDK is around 280M, and the Spark binary is 73M. Then that leaves another 636M for base + whatever. I guess I'll have to look at the imagelayers if I'm really curious.
he means all the dependencies and their dependencies. if you'd like to tackle shrinking the spark image, please open an issue for it. we've done some work on this and you can use the example investigation above to help improve image sizes.


Question: // GetHTTPCodeFromTestContainer executes a curl via kubectl exec in a test container and returns the status code. func (config *NetworkingTestConfig) GetHTTPCodeFromTestContainer(ctx context.Context, path, targetIP string, targetPort int) (int, error) { <del> cmd := fmt.Sprintf("curl -g -q -s -o /dev/null -w %%{http_code} http://%s:%d%s", targetIP, targetPort, </del> <ins> cmd := fmt.Sprintf("curl -g -q -s -o /dev/null -w %%{http_code} http://%s%s", net.JoinHostPort(targetIP, strconv.Itoa(targetPort)), </ins> path) stdout, stderr, err := e2epod.ExecShellInPodWithFullOutput(ctx, config.f, config.TestContainerPod.Name, cmd) // We only care about the status code reported by curl,

sig-testing-kind: cloud-provider-kind (IPv6), master (dev) [non-serial] (IPv6),%20master%20(dev)%20%5Bnon-serial%5D No response /sig network cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
Checking It fails to get a value from the healtcheck nodeport however, it seems it is listening there /help /sig network you were recently working on this area of healthcecks and ip families, does this ring a bell? This should be easy to reproduce using kind and cloud-provider-kind if someone wants to help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Checking however, it seems it is listening there /sig network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
it is not a flake, curl from test container fails because of bad url formatting (for ipv6 hosts).
embarrassing, now I can see it was obvious


Question: if [ $? -eq 0 ]; then if ! which nsenter > /dev/null; then echo "Do not find nsenter. Install it." <del> mkdir -p /tmp/nsenter-install cd /tmp/nsenter-install curl https://www.kernel.org/pub/linux/utils/util-linux/v2.24/util-linux-2.24.tar.gz | tar -zxf- </del> <ins> NSENTER_BUILD_DIR=$(mktemp -d /tmp/nsenter-build-XXXXXX) cd $NSENTER_BUILD_DIR curl https://www.kernel.org/pub/linux/utils/util-linux/v2.31/util-linux-2.31.tar.gz | tar -zxf- </ins> sudo apt-get update sudo apt-get --yes install make sudo apt-get --yes install gcc <del> cd util-linux-2.24 </del> <ins> cd util-linux-2.31 </ins> ./configure --without-ncurses make nsenter sudo cp nsenter /usr/local/bin <del> rm -rf /tmp/nsenter-install </del> <ins> rm -rf $NSENTER_BUILD_DIR </ins> fi fi

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Code inspection caused me to notice is installing downloading an ancient util-linux in order to build nsenter in a hard coded directory under /tmp. What you expected to happen: A more current version of util-linux would be download into a /tmp directory with a securely generated name. How to reproduce it (as minimally and precisely as possible): Read the section of marked "Install nsenter for ubuntu images". Anything else we need to know?: It's easy enough to use mktemp and download a newer util-linux and I have patches to do so. An alternative might be to do something like with nsenter via a container, but jpetazzo's container is also not actively curated and is using an old version of util-linux too. It's probably not appropriate to simply calculate the latest util-linux and download it, versus a specific one by version. But that means a steady stream of tweaks to update the version in use as new util-linux come out and are demonstrated to still be compatible. Would be a nice place to add CI as Trusty's LTS with a couple more years of support so this issue is unfortunately likely to hang around. Environment: OS (e.g. from /etc/os-release): Ubuntu Trusty Tahr
/sig testing
/assign


Question: Mem_Buf_Limit    5MB Skip_Long_Lines  On Refresh_Interval 5 <del> DB               flb_kube.db </del> <ins> DB               flb_kube.db </ins> <del> # Settings from fluentd missing here. </del> <ins> # Settings from fluentd missing here. </ins> # tag reform.* # format json # time_key time

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There's a race condition between starting Windows service and when the VMs network is actually available to communicate with the VM's metadata server. If the service starts first and cannot communicate with the metadata server or start talking to the stackdriver logging service it'll crash and not resume because the service is not configured to restart on failure. Typically a is used for a service like this but Windows does not support privileged containers so a standard Windows Service is used. What you expected to happen: If the service fails to start it should be automatically be restarted. How to reproduce it (as minimally and precisely as possible): This is a flaky issue but a way to repro this is to effectively by RDP'ing into the node and running: Anything else we need to know?: Environment: Kubernetes version (use ): v1.16 and onward Cloud provider or hardware configuration: GCE OS (e.g: ): Windows Kernel (e.g. ): Windows (any version) Install tools: cluster/gce/windows/README-GCE-Windows-kube- Network plugin and version (if this is a network-related bug): Others:
/sig windows
/assign jeremyedwards
GitHub didn't allow me to assign the following users: jeremyedwards. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign jeremyedwards Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details