Question: } waiting = false } else if !inspectResult.State.FinishedAt.IsZero() { <del> // TODO(dchen1107): Integrate with event to provide a better reason </del> <ins> reason := "" // Note: An application might handle OOMKilled gracefully. // In that case, the container is oom killed, but the exit // code could be 0. if inspectResult.State.OOMKilled { reason = "OOM Killed" } else { reason = inspectResult.State.Error } </ins> containerStatus.State.Termination = &api.ContainerStateTerminated{ ExitCode:   inspectResult.State.ExitCode, <del> Reason:     "", </del> <ins> Reason:     reason, </ins> StartedAt:  util.NewTime(inspectResult.State.StartedAt), FinishedAt: util.NewTime(inspectResult.State.FinishedAt), }

docker/docker/pull/8479 was merged and should be available in docker 1.3.2. Kubelete should pick up this newly introduced state from Container status, and propagate it to upper layer.
cc/ I am working on this today. But my initial test with docker is that the feature doesn't work well. I started a container with small memory limit and saw the container was terminated by exit code 137: I ran docker inspect and there is no OOMKilled state at Container State: "State": { "ExitCode": 137, "FinishedAt": "2014-12-16T01:28:15.506537166Z", "Paused": false, "Pid": 0, "Restarting": false, "Running": false, "StartedAt": "2014-12-16T01:28:14.427052134Z" }, I checked kernel log which fill with OOM kill information: Dec 16 01:16:25 kubernetes-minion-2 kernel: [ 2393.] [<] ? pagefaultoutofmemory+0x10/0x80 Dec 16 01:16:25 kubernetes-minion-2 kernel: [ 2393.] Memory cgroup out of memory: Kill process (supervisord) score 2049 or sacrifice child Dec 16 01:16:35 kubernetes-minion-2 kernel: [ 2403.] [<] ? pagefaultoutofmemory+0x10/0x80 Dec 16 01:16:35 kubernetes-minion-2 kernel: [ 2403.] Memory cgroup out of memory: Kill process (supervisord) score 2051 or sacrifice child Dec 16 01:16:45 kubernetes-minion-2 kernel: [ 2413.] [<] ? pagefaultoutofmemory+0x10/0x80 Dec 16 01:16:45 kubernetes-minion-2 kernel: [ 2413.] Memory cgroup out of memory: Kill process (supervisord) score 2062 or sacrifice child Dec 16 01:16:55 kubernetes-minion-2 kernel: [ 2423.] [<] ?

pagefaultoutofmemory+0x10/0x80 Dec 16 01:16:55 kubernetes-minion-2 kernel: [ 2423.] Memory cgroup out of memory: Kill process (supervisord) score 2051 or sacrifice child Dec 16 01:17:05 kubernetes-minion-2 kernel: [ 2433.] [<] ? pagefaultoutofmemory+0x10/0x80 Dec 16 01:17:05 kubernetes-minion-2 kernel: [ 2433.] Memory cgroup out of memory: Kill process (supervisord) score 2036 or sacrifice child Dec 16 01:17:15 kubernetes-minion-2 kernel: [ 2443.] [<] ? pagefaultoutofmemory+0x10/0x80 Dec 16 01:17:15 kubernetes-minion-2 kernel: [ 2443.] Memory cgroup out of memory: Kill process (supervisord) score 2074 or sacrifice child Dec 16 01:17:25 kubernetes-minion-2 kernel: [ 2453.] [<] ? pagefaultoutof_memory+0x10/0x80 The docker I am running is 1.3.3 Which version should include your PR,
The latest version '1.4' has my OOM patches.
This is what I thought, and thanks for confirmation.
may as well update to 1.4, no reason not to do so, we'll get a few weeks of soak before we cut the next release... , Dawn Chen wrote:
I am doing some testing on 1.4 now. I am running into some issues with it, and debugging it. If it is accepted, I will upgrade OSS to docker 1.4, then ask to build new containervm-image later.


Question: } var defaultPriorityQueueOptions = priorityQueueOptions{ <del> clock:                             util.RealClock{}, </del> <ins> clock:                             clock.RealClock{}, </ins> podInitialBackoffDuration:         DefaultPodInitialBackoffDuration, podMaxBackoffDuration:             DefaultPodMaxBackoffDuration, podMaxInUnschedulablePodsDuration: DefaultPodMaxInUnschedulablePodsDuration,

We should use /kind cleanup
/remove-kind feature /good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-kind feature /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /triage accepted
/assign


Question: runtimeService internalapi.RuntimeService, containerManager cm.ContainerManager) *HollowKubelet { d := &kubelet.Dependencies{ <del> KubeClient:           client, HeartbeatClient:      heartbeatClient, ProbeManager:         probetest.FakeManager{}, RemoteRuntimeService: runtimeService, RemoteImageService:   imageService, CAdvisorInterface:    cadvisorInterface, Cloud:                nil, OSInterface:          &containertest.FakeOS{}, ContainerManager:     containerManager, VolumePlugins:        volumePlugins(), TLSOptions:           nil, OOMAdjuster:          oom.NewFakeOOMAdjuster(), Mounter:              &mount.FakeMounter{}, Subpather:            &subpath.FakeSubpath{}, HostUtil:             hostutil.NewFakeHostUtil(nil), </del> <ins> KubeClient:               client, HeartbeatClient:          heartbeatClient, ProbeManager:             probetest.FakeManager{}, RemoteRuntimeService:     runtimeService, RemoteImageService:       imageService, CAdvisorInterface:        cadvisorInterface, Cloud:                    nil, OSInterface:              &containertest.FakeOS{}, ContainerManager:         containerManager, VolumePlugins:            volumePlugins(), TLSOptions:               nil, OOMAdjuster:              oom.NewFakeOOMAdjuster(), Mounter:                  &mount.FakeMounter{}, Subpather:                &subpath.FakeSubpath{}, HostUtil:                 hostutil.NewFakeHostUtil(nil), PodStartupLatencyTracker: kubeletutil.NewPodStartupLatencyTracker(), </ins> } return &HollowKubelet{

All kubemark tests started failing around 11-04. https://k8s- https://k8s- The testing framework is not able to schedule pods. hollow kubelets has errors like: Finally (when the test already failed) it changes to Suspected commit range: ci-kubernetes-kubemark-500-gce.Overall 11-04 https://k8s- No response No response /sig scalability
/triage accepted
/assign
None of the commits in the range looks indeed suspicious (mostly some endpointslice and metrics changes).
I forgot that the commit in the testgrid doesn't match the tested version. The real suspected range is Probably is a suspect.
It helped !


Question: func kubeconfigClientConfig(s *options.KubeletServer) (*restclient.Config, error) { return clientcmd.NewNonInteractiveDeferredLoadingClientConfig( <del> &clientcmd.ClientConfigLoadingRules{ExplicitPath: s.KubeConfig.Value()}, </del> <ins> &clientcmd.ClientConfigLoadingRules{ExplicitPath: s.KubeConfig}, </ins> &clientcmd.ConfigOverrides{}, ).ClientConfig() } // createClientConfig creates a client configuration from the command line arguments. <del> // If --kubeconfig is explicitly set, it will be used. If it is not set but // --require-kubeconfig=true, we attempt to load the default kubeconfig file. </del> <ins> // If --kubeconfig is explicitly set, it will be used. </ins> func createClientConfig(s *options.KubeletServer) (*restclient.Config, error) { <del> // If --kubeconfig was not provided, it will have a default path set in cmd/kubelet/app/options/options.go. // We only use that default path when --require-kubeconfig=true. The default path is temporary until --require-kubeconfig is removed. // TODO(#41161:v1.10.0): Remove the default kubeconfig path and --require-kubeconfig. if s.BootstrapKubeconfig != "" || s.KubeConfig.Provided() || s.RequireKubeConfig == true { </del> <ins> if s.BootstrapKubeconfig != "" || len(s.KubeConfig) > 0 { </ins> return kubeconfigClientConfig(s) } else { return nil, fmt.Errorf("createClientConfig called in standalone mode")

This will be the end of the deprecation period for --require-kubeconfig.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
ref
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: glog.V(5).Infof("Statfs_t of %v: %+v", path, buf) if buf.Type == linuxTmpfsMagic { return mediumMemory, !notMnt, nil <del> } else if buf.Type == linuxHugetlbfsMagic { </del> <ins> } else if int64(buf.Type) == linuxHugetlbfsMagic { </ins> return mediumHugepages, !notMnt, nil } return mediumUnknown, !notMnt, nil

https://k8s- https://k8s- cc: /kind bug /priority failing-test
Fix in progress in


Question: case testpatterns.DynamicPV: By("Deleting pvc") // We only delete the PVC so that PV (and disk) can be cleaned up by dynamic provisioner <del> if r.pv.Spec.PersistentVolumeReclaimPolicy != v1.PersistentVolumeReclaimDelete { </del> <ins> if r.pv != nil && r.pv.Spec.PersistentVolumeReclaimPolicy != v1.PersistentVolumeReclaimDelete { </ins> framework.Failf("Test framework does not currently support Dynamically Provisioned Persistent Volume %v specified with reclaim policy that isnt %v", r.pv.Name, v1.PersistentVolumeReclaimDelete) } <del> err := framework.DeletePersistentVolumeClaim(f.ClientSet, r.pvc.Name, f.Namespace.Name) framework.ExpectNoError(err, "Failed to delete PVC %v", r.pvc.Name) err = framework.WaitForPersistentVolumeDeleted(f.ClientSet, r.pv.Name, 5*time.Second, 5*time.Minute) framework.ExpectNoError(err, "Persistent Volume %v not deleted by dynamic provisioner", r.pv.Name) </del> <ins> if r.pvc != nil { err := framework.DeletePersistentVolumeClaim(f.ClientSet, r.pvc.Name, f.Namespace.Name) framework.ExpectNoError(err, "Failed to delete PVC %v", r.pvc.Name) if r.pv != nil { err = framework.WaitForPersistentVolumeDeleted(f.ClientSet, r.pv.Name, 5*time.Second, 5*time.Minute) framework.ExpectNoError(err, "Persistent Volume %v not deleted by dynamic provisioner", r.pv.Name) } } </ins> default: framework.Failf("Found PVC (%v) or PV (%v) but not running Preprovisioned or Dynamic test pattern", r.pvc, r.pv) }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce Which test(s) are failing: I0221 20:48:18.462] [sig-storage] In-tree Volumes I0221 20:48:18.462] [ I0221 20:48:18.462] [Driver: gcepd] I0221 20:48:18.462] [ I0221 20:48:18.462] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode I0221 20:48:18.463] [ I0221 20:48:18.463] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources Since when has it been failing: unsure Testgrid link: Reason for failure: From the logs, it looks like it's deleting both PVC and PV objects at the same time, which is not correct. I0221 20:48:18.460] Feb 21 20:48:09.866: INFO: Deleting PersistentVolumeClaim "pvc-cjv2l" I0221 20:48:18.460] Feb 21 20:48:09.923: INFO: Deleting PersistentVolume "pvc--3619-11e9-b923-" should be changed to only delete PVC and wait for PV to be deleted by the driver. Can it just call which does it correctly? Anything else we need to know:
cc
It has been like that already before my recent code refactoring: Therefore I have to defer to - I myself don't really know what this test does or why it was written like that.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
/priority critical-urgent


Question: if len(defaultClasses) == 0 { return nil, nil } <ins> // Primary sort by creation timestamp, newest first // Secondary sort by class name, ascending order sort.Slice(defaultClasses, func(i, j int) bool { if defaultClasses[i].CreationTimestamp.UnixNano() == defaultClasses[j].CreationTimestamp.UnixNano() { return defaultClasses[i].Name < defaultClasses[j].Name } return defaultClasses[i].CreationTimestamp.UnixNano() > defaultClasses[j].CreationTimestamp.UnixNano() }) </ins> if len(defaultClasses) > 1 { <del> klog.V(4).Infof("GetDefaultClass %d defaults found", len(defaultClasses)) return nil, errors.NewInternalError(fmt.Errorf("%d default StorageClasses were found", len(defaultClasses))) </del> <ins> klog.V(4).Infof("%d default StorageClasses were found, choosing the newest: %s", len(defaultClasses), defaultClasses[0].Name) </ins> } return defaultClasses[0], nil }

Presumably this would occur when changing the default from one class to another. E.g. Class X exists and is marked default Class Y is created Class Y is marked as default Uh oh!! 2 defaults Class X is marked as not default At the case in point, when there are 2 defaults, we currently do not assign default values to new PVCs. We should change this semantic - while there are two, pick one randomly or heuristically. It's still correct. Example: Suppose that you have one user changing the default and another user creating a PVC at exactly the same time. What happens? Depending on network, CPU scheduling, memory allocations, (etc - totally non-deterministic) the API server might see either of those operations first. The PVC creator will either win the race and get the old default class or it will lose the race and get the new default class. The difference could literally be whose network has a faster path to the API server. My point is that whichever one you pick, as long as it is one of those two defaults, it is a correct choice! Any other FooClass things we do should probably change, too. I think that's IngressClass and RuntimeClass.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
picking either sounds reasonable but what if we allow only one default to be set and disallow/warn users if they try to create another default.

I can think of issues with this too, mainly when switching over to a new default by removing an existing class A and then setting B as a default but managing that orphaned period while the transition happens, maybe we can have the heuristic there? I'm sure I'm missing something here.
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There is a problem when there is no default storage class for a while and another user creates a PVC. Such PVC won't get the new SC. We're trying to fix it in Until then, having two default storage classes for a brief moment is IMO slightly better, because at least users get an error message and can retry PVC creation. In addition, (in-tree) admission plugin in HA environment could be racy, as different API servers are admitting updates of different StorageClasses in parallel.
/assign
I wouldn't make 2 an error - just pick one.
Ok, I'd take a shot at this. Thanks for clearing things up.
+1 Still, having two default classes for a longer time is IMO bad, there should be a metric + alert if there are 2 or more default SCs for X minutes.
it's not ideal - a metric seems OK
hello there, im new to open source. i don't know how to contribute. can anyone hep me please?
I see this issue was labeled as "good first issue" and I would like to start working on it!
I see there is a pr related to this issue , are you still working on this?
Is this issue still open?
/assign
Yes, there's already a . It's linked to the issue itself.
The PR is specifically for StorageClass, but we also need to audit other FooClass things - they should probably change, too. I think that's IngressClass and RuntimeClass. Someone can open new PRs for these.
Oh, I see the second PR, too.

Will look at them ASAP
/remove-sig network
Do we emit a response header when someone makes a request that ends up setting two IngressClasses as default?
that seems a nice thing to do, I didn't know you can add warnings on admissions plugins but seems implemented it here
Is this issue still open??
StorageClass implementation is in I am not sure about those. I'd suggest opening separate issues. /close
Closing this issue. <detailsIn response to : whichever one you pick, as long as it is one of those two defaults, it is a correct choice! </details


Question: logger.Error(err, "Scheduler cache AddPod failed", "pod", klog.KObj(pod)) } <del> sched.SchedulingQueue.AssignedPodAdded(logger, pod) </del> <ins> // SchedulingQueue.AssignedPodAdded has a problem: // It internally pre-filters Pods to move to activeQ, // while taking only in-tree plugins into consideration. // Consequently, if custom plugins that subscribes Pod/Add events reject Pods, // those Pods will never be requeued to activeQ by an assigned Pod related events, // and they may be stuck in unschedulableQ. // // Here we use MoveAllToActiveOrBackoffQueue only when QueueingHint is enabled. // (We cannot switch to MoveAllToActiveOrBackoffQueue right away because of throughput concern.) if utilfeature.DefaultFeatureGate.Enabled(features.SchedulerQueueingHints) { sched.SchedulingQueue.MoveAllToActiveOrBackoffQueue(logger, queue.AssignedPodAdd, nil, pod, nil) } else { sched.SchedulingQueue.AssignedPodAdded(logger, pod) } </ins> } func (sched *Scheduler) updatePodInCache(oldObj, newObj interface{}) {

/sig scheduling /kind cleanup Ref: A test helper function creates a goroutine to pop a Pod from the scheduling queue. This goroutine never stops until the scheduling queue is ed or is somehow stopped for waiting for a new Pod. Ideally, when reaching timeout, we should somehow stop from waiting for Pod to be enqueued to activeQ.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We can replace this function with and it should be enough? I think we no longer need to maintain this function.
Not enough, the core of this problem is that we cannot cancel once we start . Even if we change function to whatever, we will hit this problem. So, we need:
Or, alternatively and much more easily, maybe we can just replace all with and remove . kills the entire test if reaching timeout and we could care less about leaked goroutine then.
I don't think we can do this. From the test case: The expected behavior of this case is that we receive a timeout to make sure no pods are to . So, reaching timeout is acutally a success. So, I think: option 1: using sth like this in : option2: do sth in the test code This part is where it confused me. The current code is fine as we closed schdulerQueue after called , and after I think there is no test case using this this function anymore(seeems that you refactored the test), so are we trying to solve a potential problem?
Oh, then we can just remove NextPod in the PR.
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

/sig scheduling /kind bug AssignedPodAdded(): AssignedPodUpdated(): When assigned Pods are created/updated, cluster events (Pod/Add or Pod/Update) should be delivered for all plugins that subscribe those events. But, actually pre-filters Pods to receive those events based on Pods' affinity and . - meaning only PodAffinity/NodeResourceFit receive them actually. The problem here is similar to - in in-tree plugins, PodAffinity, PodTopologySpread and NodeResourceFit subscribe Pod add and/or update events. And, Pods rejected by PodTopologySpread will never be requeued to activeQ by Pod-related events. The same happens for custom plugins registering pod add and/or update events. All Pods will get chances to get back to activeQ by Pod-related events. Run ↓ in No response <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: "k8s.io/kubernetes/pkg/kubelet/pluginmanager/cache" "k8s.io/kubernetes/pkg/kubelet/stats/pidlimit" "k8s.io/kubernetes/pkg/kubelet/status" <ins> "k8s.io/kubernetes/pkg/kubelet/userns/inuserns" </ins> schedulerframework "k8s.io/kubernetes/pkg/scheduler/framework" "k8s.io/kubernetes/pkg/util/oom" )

is no longer compilable on macOS in the master branch. It was compilable with Kubernetes v1.29.0 and v1.30.0-alpha.0. It should be compilable. No response <detailsmaster () (This is just the latest commit, not the commit that introduced the regression) </details<detailsN/A </details<detailsmacOS 14.2.1 (Intel) </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Regression in:


Question: } // Act <del> go reconciler.Run(wait.NeverStop) </del> <ins> ch := make(chan struct{}) go reconciler.Run(ch) defer close(ch) </ins> // Assert waitForNewAttacherCallCount(t, 1 /* expectedCallCount */, fakePlugin)

Looks like similar issue with
That's the PV controller this is the attach controller, thus separate issues.


Question: runtime.HandleError(fmt.Errorf("error creating forwarding stream for port %d -> %d: %v", port.Local, port.Remote, err)) return } <ins> defer pf.streamConn.RemoveStreams(dataStream) </ins> localError := make(chan struct{}) remoteDone := make(chan struct{})

The port forwarding client leaks memory when a port forwarding session is reused over time. The port forwarder client should not leak memory. forwarding a port with kubectl. and close many connections to the forwarded port. The amount of memory leaked for each connection is quite small, so many connections are needed to observe the leak. This is related to , which was reported for the kubelet and fixed in . However, the same problem happens to the client. The client needs to call on the , otherwise the streams created by will be kept in memory until the port forwarder itself is closed, even if the streams themselves closed long before. <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted
/cc


Question: local targets=( vendor/github.com/onsi/ginkgo/ginkgo test/e2e/e2e.test <ins> cluster/images/conformance/go-runner </ins> cmd/kubectl ) echo "${targets[@]}"

There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: github.com/json-iterator/go v1.1.6 github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024 github.com/karrick/godirwalk v1.7.5 // indirect <del> github.com/libopenstorage/openstorage v0.0.0-20170906232338-093a0c388875 </del> <ins> github.com/libopenstorage/openstorage v1.0.0 </ins> github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de github.com/lithammer/dedent v1.1.0 github.com/lpabon/godbc v0.1.1 // indirect

Found this when trying to see which vendored dependencies does not use a proper tag (and is using a random SHA). The dependency is very old and there have been a lot of updates to it: Looks like the However there does not seem to be anyone around to take care of the portworx plugin. Is it time to deprecate it? is anyone using it? /sig storage
cc ?
/area code-organization
Hello The portworx volume plugin is definitely in use. The portworx volume plugin doesn't need frequent updates since the plugin acts as a simple client to all provisioning calls. In the same view, the vendored openstorage package doesn't need to be updated too. The server side implementation in Portworx handles all the latest Portworx features.
cc:
in that case, can you please update the dependency to a specific tag instead of a random SHA?
/assign updated the title of the issue as well
my request is not for a new release or tag in libopenstorage/openstorage repository. Just to update the SHA we use in our k/k repository to one of the released versions ( ). This is not asking or demanding anything to be done in libopenstorage/openstorage, just that we should update what we use from there (and avoid a random SHA). We are trying to do this for all vendored libraries (one by one … slowly as part of the sig-arch code organization sub project, so it’s not a random request)
/assign
GitHub didn't allow me to assign the following users: lpabon. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After a chat with - apparently this request is coming across the wrong way I am closing this out as it seems to add a over head /close
Closing this issue. <detailsIn response to : After a chat with - apparently this request is coming across the wrong way /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func SetupSignalHandler() <-chan struct{} { close(onlyOneSignalHandler) // panics when called twice <ins> shutdownHandler = make(chan os.Signal, 2) </ins> stop := make(chan struct{}) <del> c := make(chan os.Signal, 2) signal.Notify(c, shutdownSignals...) </del> <ins> signal.Notify(shutdownHandler, shutdownSignals...) </ins> go func() { <del> <-c </del> <ins> <-shutdownHandler </ins> close(stop) <del> <-c </del> <ins> <-shutdownHandler </ins> os.Exit(1) // second signal. Exit directly. }() return stop } <ins> // RequestShutdown emulates a received event that is considered as shutdown signal (SIGTERM/SIGINT) // This returns whether a handler was notified func RequestShutdown() bool { if shutdownHandler != nil { select { case shutdownHandler <- shutdownSignals[0]: return true default: } } return false } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I configured Kubelet to run as a service on Windows, with a setting to restart on failure (). I noticed that it restarts even when I explicitly stop the service with , even though the documentation states the following (I left it at , the default): What you expected to happen: I expected the Kubelet to only restart when killed unexpectedly, per above docs. Anything else we need to know?: I think the problem might be due to this part of our service control integration, but I'm not really sure: It looks like we send and then , but the official golang example sends and just returns without exiting. (). Is this difference meaningful? Kubernetes version (use ): 1.13.2 Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Windows /cc /sig node /sig windows
The same should apply to kube-proxy as well.
i believe you implemented this functionality. can you please take a look? cc:


Question: // to the API server. // // The caller must not modify the content after the call. <del> PublishResources(ctx context.Context, resources Resources) </del> <ins> // // Returns an error if KubeClient or NodeName options were not // set in Start() to create the DRAPlugin instance. PublishResources(ctx context.Context, resources Resources) error </ins> // This unexported method ensures that we can modify the interface // without causing an API break of the package

When calling draplugin.Start without supplying kubeclient no error is reported, while the kubeclient is vital for further announcement of resourceSlice. An error needs to be returned when nodeName or kubeclient parameters are not set. Do not supply kubeclient parameter when creating draplugin, for isntance here: No response master <details</details<details</details<details</details<details</details<details</details
/sig node /triage accepted /project 95


Question: DomainNameLabel: &domainNameLabel, } } <del> pip.Tags = map[string]*string{"service": &serviceName} </del> <ins> pip.Tags = map[string]*string{ serviceTagKey:  &serviceName, clusterNameKey: &clusterName, } </ins> if az.useStandardLoadBalancer() { pip.Sku = &network.PublicIPAddressSku{ Name: network.PublicIPAddressSkuNameStandard,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We deployed two Kubernetes clusters to the same Azure Resource Group. Created services of load balancer type on both clusters. One of the services never received a public IP. Upon investigation, we noticed that the first Kubernetes cluster deleted the Azure Public IP that was created for the service by the other cluster. Apparently, all public IPs in the resource group are retrieved: then public IPs which don't match this criteria: are deleted. This logic besides the name of the kubernetes service should also consider the K8s cluster where the public IP originated from to avoid stepping on each other's toes when multiple clusters share the same resource group. What you expected to happen: In case multiple clusters run in the same resource group a cluster should delete only resources created by itself. How to reproduce it (as minimally and precisely as possible): Create two Kubernetes clusters (1.14) with different cluster names to the same resource group than create kubernetes services of load balancer type in each cluster. Anything else we need to know?: Environment: Kubernetes version (use ): 1.4 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks for reporting the issue, this is the bug we should fix for sharing resource group. /sig azure /kind bug /assign


Question: local dest_dir="${2}" export-windows-docker-event-log "${node}" <ins> export-windows-docker-images-list "${node}" </ins> local remote_files=() for file in ${windows_node_logfiles[@]}; do remote_files+=( "${WINDOWS_LOGS_DIR}${file}" )

When a test run completes it would be useful to have a list of all the test containers that were pulled onto the node - see for the motivation. Let's add a call to at - just after seems like it should work fine. We can dump the output to a file and gather it from each node just like we do for
/assign Yang, we should also consider adding to the log gathering that the GCE diagnostics tool does :)
/sig windows
/reopen in the of our on 03-22 I don't see in the artifacts for the Windows node. Can you check on this? Did we miss a step or is something failing?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Oh, missed to add the new log file to collector list.
Added the fix:


Question: } // Compute a three way strategic merge patch to send to server. <del> patch, err := strategicpatch.CreateThreeWayMergePatch(original, modified, current, versionedObject, true) </del> <ins> patch, err := strategicpatch.CreateThreeWayMergePatch(original, modified, current, versionedObject, p.overwrite) </ins> if err != nil { format := "creating patch with:noriginal:n%snmodified:n%sncurrent:n%snfor:" return nil, cmdutil.AddSourceToErr(fmt.Sprintf(format, original, modified, current), source, err)

This issue tracks the implementation of the --overwrite flag in kubectl apply, per in .
Changed the title to address only the --overwrite flag, since deals with the --force flag.
progress on this issue is blocked waiting for to merge.


Question: <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="223.442,387.264 218.371,377.3 223.389,379.764 228.371,377.229 "/> </g> <g> <del> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 606 581.25 C 636.898,581.25 679.302,322 711.464,322"/> <polygon style="fill: #000000" points="606,576.25 596,581.25 606,586.25 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="606,576.25 596,581.25 606,586.25 "/> <polygon style="fill: #000000" points="718.964,322 708.964,327 711.464,322 708.964,317 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="718.964,322 708.964,327 711.464,322 708.964,317 "/> </del> <ins> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 319.892 503.5 C 392.964,503.5 639.13,244.5 713.464,244.5"/> <polygon style="fill: #000000" points="319.892,498.5 309.892,503.5 319.892,508.5 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="319.892,498.5 309.892,503.5 319.892,508.5 "/> <polygon style="fill: #000000" points="720.964,244.5 710.964,249.5 713.464,244.5 710.964,239.5 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="720.964,244.5 710.964,249.5 713.464,244.5 710.964,239.5 "/> </g> <g> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="93.65" y1="503.5" x2="126.981" y2="503.5"/> <polygon style="fill: #000000" points="134.481,503.5 124.481,508.5 126.981,503.5 124.481,498.5 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="134.481,503.5 124.481,508.5 126.981,503.5 124.481,498.5 "/> </g> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="221.612" y="111.49"> <tspan x="221.612" y="111.49"></tspan> </text> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="1209" y="339.5"> <tspan x="1209" y="339.5">docker</tspan> </text> <g> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="793.753" y1="272.636" x2="968.266" y2="363.6"/> <polygon style="fill: #000000" points="974.917,367.066 963.738,366.878 968.266,363.6 968.361,358.01 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="974.917,367.066 963.738,366.878 968.266,363.6 968.361,358.01 "/> </g> <text font-size="12.7998" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="978" y="434.5"> <tspan x="978" y="434.5">..</tspan> </text> <text font-size="27.0929" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="1067" y="437"> <tspan x="1067" y="437">...</tspan> </text> <g> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="792.8" y1="273" x2="1173.14" y2="365.792"/> <polygon style="fill: #000000" points="1180.43,367.57 1169.53,370.057 1173.14,365.792 1171.9,360.342 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="1180.43,367.57 1169.53,370.057 1173.14,365.792 1171.9,360.342 "/> </ins> </g> <g> <del> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 606 641.75 C 636.898,641.75 679.302,717.938 711.464,717.938"/> <polygon style="fill: #000000" points="606,636.75 596,641.75 606,646.75 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="606,636.75 596,641.75 606,646.75 "/> <polygon style="fill: #000000" points="718.964,717.938 708.964,722.938 711.464,717.938 708.964,712.938 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="718.964,717.938 708.964,722.938 711.464,717.938 708.964,712.938 "/> </del> <ins> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="792.8" y1="273" x2="794.057" y2="358.365"/> <polygon style="fill: #000000" points="794.167,365.864 789.02,355.939 794.057,358.365 799.019,355.792 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="794.167,365.864 789.02,355.939 794.057,358.365 799.019,355.792 "/> </ins> </g> <ins> <text font-size="12.7998" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="720" y="220"> <tspan x="720" y="220"></tspan> </text> </ins> <g> <del> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 790.8 708.438 C 825,677.75 879,674.75 916.908,702.665"/> <polygon style="fill: #000000" points="922.947,707.112 911.93,705.209 916.908,702.665 917.86,697.156 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="922.947,707.112 911.93,705.209 916.908,702.665 917.86,697.156 "/> </del> <ins> <rect style="fill: #ffffff" x="660" y="660" width="630" height="381"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="660" y="660" width="630" height="381"/> </ins> </g> <g> <del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="801.8" y1="727.438" x2="811.94" y2="727.438"/> <polygon style="fill: #000000" points="801.8,722.438 791.8,727.438 801.8,732.438 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="801.8,722.438 791.8,727.438 801.8,732.438 "/> <polygon style="fill: #000000" points="819.44,727.438 809.44,732.438 811.94,727.438 809.44,722.438 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="819.44,727.438 809.44,732.438 811.94,727.438 809.44,722.438 "/> </del> <ins> <rect style="fill: #ffffff" x="686" y="789" width="580" height="227"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="686" y="789" width="580" height="227"/> </ins> </g> <ins> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="685" y="692"> <tspan x="685" y="692">Node</tspan> </text> </ins> <g> <del> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 934.594 747.007 C 928,762 804,738 797.63,754.926"/> <polygon style="fill: #000000" points="939.171,749.02 938.619,737.853 930.017,744.994 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="939.171,749.02 938.619,737.853 930.017,744.994 "/> <polygon style="fill: #000000" points="794.988,761.945 793.831,750.825 797.63,754.926 803.19,754.347 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="794.988,761.945 793.831,750.825 797.63,754.926 803.19,754.347 "/> </del> <ins> <rect style="fill: #ffffff" x="721.2" y="703" width="69.6" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="721.2" y="703" width="69.6" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="756" y="725.9"> <tspan x="756" y="725.9">kubelet</tspan> </text> </ins> </g> <g> <del> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 974.333 745.616 C 998,776 1131,714 1154.77,755.585"/> <polygon style="fill: #000000" points="978.278,742.544 968.188,737.727 970.389,748.689 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="978.278,742.544 968.188,737.727 970.389,748.689 "/> <polygon style="fill: #000000" points="1158.49,762.097 1149.19,755.896 1154.77,755.585 1157.87,750.934 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="1158.49,762.097 1149.19,755.896 1154.77,755.585 1157.87,750.934 "/> </del> <ins> <rect style="fill: #ffffff" x="718.2" y="836.1" width="148" height="133"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="718.2" y="836.1" width="148" height="133"/> </ins> </g> <g> <del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="960.522" y1="745.233" x2="970.506" y2="756.696"/> <polygon style="fill: #000000" points="964.293,741.949 953.955,737.692 956.752,748.517 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="964.293,741.949 953.955,737.692 956.752,748.517 "/> <polygon style="fill: #000000" points="975.431,762.352 965.093,758.095 970.506,756.696 972.634,751.527 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="975.431,762.352 965.093,758.095 970.506,756.696 972.634,751.527 "/> </del> <ins> <rect style="fill: #ffffff" x="758.55" y="906.1" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="758.55" y="906.1" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="803.2" y="929"> <tspan x="803.2" y="929">container</tspan> </text> </ins> </g> <g> <del> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 478.974 491 C 552.046,491 637.13,312.5 711.464,312.5"/> <polygon style="fill: #000000" points="478.974,486 468.974,491 478.974,496 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="478.974,486 468.974,491 478.974,496 "/> <polygon style="fill: #000000" points="718.964,312.5 708.964,317.5 711.464,312.5 708.964,307.5 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="718.964,312.5 708.964,317.5 711.464,312.5 708.964,307.5 "/> </del> <ins> <rect style="fill: #ffffff" x="747.8" y="896.2" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="747.8" y="896.2" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="792.45" y="919.1"> <tspan x="792.45" y="919.1">container</tspan> </text> </ins> </g> <g> <del> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 478.932 505.457 C 677,522.75 650,676.75 712.305,704.479"/> <polygon style="fill: #000000" points="479.367,500.476 468.97,504.587 478.497,510.438 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="479.367,500.476 468.97,504.587 478.497,510.438 "/> <polygon style="fill: #000000" points="719.157,707.529 707.988,708.031 712.305,704.479 712.054,698.895 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="719.157,707.529 707.988,708.031 712.305,704.479 712.054,698.895 "/> </del> <ins> <rect style="fill: #ffffff" x="737.4" y="886.3" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="737.4" y="886.3" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="782.05" y="909.2"> <tspan x="782.05" y="909.2">cAdvisor</tspan> </text> </ins> </g> <ins> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="792.2" y="902.6"> <tspan x="792.2" y="902.6"></tspan> </text> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="740.2" y="862.6"> <tspan x="740.2" y="862.6">Pod</tspan> </text> </ins> <g> <del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="93.65" y1="503.5" x2="126.981" y2="503.5"/> <polygon style="fill: #000000" points="134.481,503.5 124.481,508.5 126.981,503.5 124.481,498.5 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="134.481,503.5 124.481,508.5 126.981,503.5 124.481,498.5 "/> </del> <ins> <g> <rect style="fill: #ffffff" x="1106.6" y="836.1" width="148" height="133"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1106.6" y="836.1" width="148" height="133"/> </g> <g> <rect style="fill: #ffffff" x="1146.95" y="906.1" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1146.95" y="906.1" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="1191.6" y="929"> <tspan x="1191.6" y="929">container</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="1136.2" y="896.2" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1136.2" y="896.2" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="1180.85" y="919.1"> <tspan x="1180.85" y="919.1">container</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="1125.8" y="886.3" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="1125.8" y="886.3" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="1170.45" y="909.2"> <tspan x="1170.45" y="909.2">container</tspan> </text> </g> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="1180.6" y="902.6"> <tspan x="1180.6" y="902.6"></tspan> </text> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="1128.6" y="862.6"> <tspan x="1128.6" y="862.6">Pod</tspan> </text> </g> <g> <g> <rect style="fill: #ffffff" x="900.9" y="836.1" width="148" height="133"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="900.9" y="836.1" width="148" height="133"/> </g> <g> <rect style="fill: #ffffff" x="941.25" y="906.1" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="941.25" y="906.1" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="985.9" y="929"> <tspan x="985.9" y="929">container</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="930.5" y="896.2" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="930.5" y="896.2" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="975.15" y="919.1"> <tspan x="975.15" y="919.1">container</tspan> </text> </g> <g> <rect style="fill: #ffffff" x="920.1" y="886.3" width="89.3" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="920.1" y="886.3" width="89.3" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="964.75" y="909.2"> <tspan x="964.75" y="909.2">container</tspan> </text> </g> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="974.9" y="902.6"> <tspan x="974.9" y="902.6"></tspan> </text> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="922.9" y="862.6"> <tspan x="922.9" y="862.6">Pod</tspan> </text> </ins> </g> <g> <del> <rect style="fill: #ffffff" x="372.024" y="477.5" width="95.95" height="54"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="372.024" y="477.5" width="95.95" height="54"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="419.999" y="500.4"> <tspan x="419.999" y="500.4">kubelet</tspan> <tspan x="419.999" y="516.4">info service</tspan> </del> <ins> <rect style="fill: #ffffff" x="947.748" y="696" width="57.1" height="38"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="947.748" y="696" width="57.1" height="38"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="976.298" y="718.9"> <tspan x="976.298" y="718.9">Proxy</tspan> </ins> </text> </g> <del> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="221.612" y="111.49"> <tspan x="221.612" y="111.49"></tspan> </del> <ins> <g> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="756" y1="741" x2="780.332" y2="876.717"/> <polygon style="fill: #000000" points="781.655,884.099 774.969,875.138 780.332,876.717 784.812,873.374 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="781.655,884.099 774.969,875.138 780.332,876.717 784.812,873.374 "/> </g> <g> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="976.298" y1="734" x2="975.033" y2="826.365"/> <polygon style="fill: #000000" points="974.931,833.864 970.068,823.797 975.033,826.365 980.067,823.934 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="974.931,833.864 970.068,823.797 975.033,826.365 980.067,823.934 "/> </g> <g> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="990.573" y1="734" x2="1172.02" y2="831.492"/> <polygon style="fill: #000000" points="1178.63,835.042 1167.45,834.713 1172.02,831.492 1172.19,825.904 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="1178.63,835.042 1167.45,834.713 1172.02,831.492 1172.19,825.904 "/> </g> <text font-size="12.8" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="1207" y="807.5"> <tspan x="1207" y="807.5">docker</tspan> </text> <g> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="791.753" y1="740.636" x2="966.266" y2="831.6"/> <polygon style="fill: #000000" points="972.917,835.066 961.738,834.878 966.266,831.6 966.361,826.01 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="972.917,835.066 961.738,834.878 966.266,831.6 966.361,826.01 "/> </g> <text font-size="12.7998" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="976" y="902.5"> <tspan x="976" y="902.5">..</tspan> </text> <text font-size="27.0929" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="1065" y="905"> <tspan x="1065" y="905">...</tspan> </ins> </text> <ins> <g> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="790.8" y1="741" x2="1171.14" y2="833.792"/> <polygon style="fill: #000000" points="1178.43,835.57 1167.53,838.057 1171.14,833.792 1169.9,828.342 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="1178.43,835.57 1167.53,838.057 1171.14,833.792 1169.9,828.342 "/> </g> <g> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="790.8" y1="741" x2="792.057" y2="826.365"/> <polygon style="fill: #000000" points="792.167,833.864 787.02,823.939 792.057,826.365 797.019,823.792 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="792.167,833.864 787.02,823.939 792.057,826.365 797.019,823.792 "/> </g> <text font-size="12.7998" style="fill: #000000;text-anchor:start;font-family:sans-serif;font-style:normal;font-weight:normal" x="718" y="688"> <tspan x="718" y="688"></tspan> </text> <g> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 319.892 503.5 C 392.964,503.5 575.93,850.5 650.264,850.5"/> <polygon style="fill: #000000" points="319.892,498.5 309.892,503.5 319.892,508.5 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="319.892,498.5 309.892,503.5 319.892,508.5 "/> <polygon style="fill: #000000" points="657.764,850.5 647.764,855.5 650.264,850.5 647.764,845.5 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="657.764,850.5 647.764,855.5 650.264,850.5 647.764,845.5 "/> </g> <g> <rect style="fill: #ffffff" x="418.774" y="551" width="176.225" height="121"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="418.774" y="551" width="176.225" height="121"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="506.886" y="583.4"> <tspan x="506.886" y="583.4">Distributed</tspan> <tspan x="506.886" y="599.4">Watchable</tspan> <tspan x="506.886" y="615.4">Storage</tspan> <tspan x="506.886" y="631.4"></tspan> <tspan x="506.886" y="647.4">(implemented via etcd)</tspan> </text> </g> </ins> </svg>

CA Advisor runs inside docker containers, but I don't believe it's correct to show the Kubelet and Proxy processes as inside the docker box on the minion. Also, shouldn't the internet-firewall-really load balance across minion proxies instead of be connected to just one SPOF minion, treating that connection more like a logical connection across all minion proxies?
This is the desired state, which is admittedly not the current state. Yeah, this is a good point. k8s currently relies on its environment for load balancing. Maybe or can speak to the desired state here.
We should just review the diagram and make it correctly reflect the state of 1.0.


Question: 'roles:kubernetes-pool': - match: grain - docker <ins> {% if grains['cloud'] is defined and grains['cloud'] == 'azure' %} - openvpn-client {% else %} - sdn {% endif %} </ins> - kubelet - kube-proxy {% if pillar.get('enable_node_monitoring', '').lower() == 'true' %}

On latest pull, the Vagrant kube- scripts completes successfully, but the Salt setup in the minion keeps going forever. End result is a cluster that is not useable. Looking at '/var/log/salt/minion' on the minion box, see lots of entries like: 2015-03-04 20:09:11,952 [ ][INFO ] Starting a new job with PID 2015-03-04 20:09:11,954 [ ][INFO ] Returning information for job: 2015-03-04 20:09:12,027 [ ][DEBUG ] Decrypting the current master AES key 2015-03-04 20:09:12,028 [ ][DEBUG ] Loaded minion key: 2015-03-04 20:09:12,217 [ ][INFO ] User vagrant Executing command with jid 2015-03-04 20:09:12,217 [ ][DEBUG ] Command details {'tgt_type': 'glob', 'jid': '', 'tgt': '*', 'ret': '', 'user': 'vagrant', 'arg': [], 'fun': ''} No other help looking at the master's salt log. This is Vagrant 1.7.2, with VirtualBox 4.3.20 on Mac OSX (Yosemite).
Is this repeatable for you if kube-down and kube-up?
Yes, this is reproducible on my laptop. But, I was able to get it up by doing a kube-push after the failed kube-up. , Eric Tune wrote:
Self assigning and will try to reproduce. Sent from my iPhone
I was able to reproduce. It appears that the Kubelet fails to start properly on first start:
The push basically caused the Kubelet to restart, and then it does work. Without the push, systemd would have eventually restarted the Kubelet. We need more investigation as to what was causing the Kubelet to fail on initial startup.


Question: package stats import ( <del> "fmt" </del> "time" <del> "github.com/Microsoft/hcsshim" </del> <ins> "k8s.io/klog/v2" "github.com/Microsoft/hcsshim" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <del> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/klog/v2" </del> statsapi "k8s.io/kubelet/pkg/apis/stats/v1alpha1" ) <ins> // windowsNetworkStatsProvider creates an interface that allows for testing the logic without needing to create a container type windowsNetworkStatsProvider interface { HNSListEndpointRequest() ([]hcsshim.HNSEndpoint, error) GetHNSEndpointStats(endpointName string) (*hcsshim.HNSEndpointStats, error) } // networkStats exposes the required functionality for hcsshim in this scenario type networkStats struct{} func (s networkStats) HNSListEndpointRequest() ([]hcsshim.HNSEndpoint, error) { return hcsshim.HNSListEndpointRequest() } func (s networkStats) GetHNSEndpointStats(endpointName string) (*hcsshim.HNSEndpointStats, error) { return hcsshim.GetHNSEndpointStats(endpointName) } </ins> // listContainerNetworkStats returns the network stats of all the running containers. func (p *criStatsProvider) listContainerNetworkStats() (map[string]*statsapi.NetworkStats, error) { <del> containers, err := hcsshim.GetContainers(hcsshim.ComputeSystemQuery{ Types: []string{"Container"}, }) </del> <ins> networkStatsProvider := newNetworkStatsProvider(p) endpoints, err := networkStatsProvider.HNSListEndpointRequest() </ins> if err != nil { <ins> klog.ErrorS(err, "Failed to fetch current HNS endpoints") </ins> return nil, err } <del> stats := make(map[string]*statsapi.NetworkStats) for _, c := range containers { cstats, err := fetchContainerStats(c) </del> <ins> networkStats := make(map[string]*statsapi.NetworkStats) for _, endpoint := range endpoints { endpointStats, err := networkStatsProvider.GetHNSEndpointStats(endpoint.Id) </ins> if err != nil { <del> klog.V(4).InfoS("Failed to fetch statistics for container, continue to get stats for other containers", "containerID", c.ID, "err", err) </del> <ins> klog.V(2).InfoS("Failed to fetch statistics for endpoint, continue to get stats for other endpoints", "endpointId", endpoint.Id, "containers", endpoint.SharedContainers) </ins> continue } <del> if len(cstats.Network) > 0 { stats[c.ID] = hcsStatsToNetworkStats(cstats.Timestamp, cstats.Network) } } return stats, nil } func fetchContainerStats(c hcsshim.ContainerProperties) (stats hcsshim.Statistics, err error) { var ( container hcsshim.Container ) container, err = hcsshim.OpenContainer(c.ID) if err != nil { return } defer func() { if closeErr := container.Close(); closeErr != nil { if err != nil { err = fmt.Errorf("failed to close container after error %v; close error: %v", err, closeErr) } else { err = closeErr </del> <ins> // only add the interface for each container if not already in the list for _, cId := range endpoint.SharedContainers { networkStat, found := networkStats[cId] if found && networkStat.Name != endpoint.Name { iStat := hcsStatToInterfaceStat(endpointStats, endpoint.Name) networkStat.Interfaces = append(networkStat.Interfaces, iStat) continue </ins> } <ins> networkStats[cId] = hcsStatsToNetworkStats(p.clock.Now(), endpointStats, endpoint.Name) </ins> } <del> }() </del> <ins> } </ins> <del> return container.Statistics() </del> <ins> return networkStats, nil </ins> } // hcsStatsToNetworkStats converts hcsshim.Statistics.Network to statsapi.NetworkStats <del> func hcsStatsToNetworkStats(timestamp time.Time, hcsStats []hcsshim.NetworkStats) *statsapi.NetworkStats { </del> <ins> func hcsStatsToNetworkStats(timestamp time.Time, hcsStats *hcsshim.HNSEndpointStats, endpointName string) *statsapi.NetworkStats { </ins> result := &statsapi.NetworkStats{ Time:       metav1.NewTime(timestamp), Interfaces: make([]statsapi.InterfaceStats, 0), } <del> adapters := sets.NewString() for _, stat := range hcsStats { iStat, err := hcsStatsToInterfaceStats(stat) if err != nil { klog.InfoS("Failed to get HNS endpoint, continue to get stats for other endpoints", "endpointID", stat.EndpointId, "err", err) continue } // Only count each adapter once. if adapters.Has(iStat.Name) { continue } result.Interfaces = append(result.Interfaces, *iStat) adapters.Insert(iStat.Name) } </del> <ins> iStat := hcsStatToInterfaceStat(hcsStats, endpointName) </ins> <del> // TODO(feiskyer): add support of multiple interfaces for getting default interface. if len(result.Interfaces) > 0 { result.InterfaceStats = result.Interfaces[0] } </del> <ins> // TODO: add support of multiple interfaces for getting default interface. result.Interfaces = append(result.Interfaces, iStat) result.InterfaceStats = iStat </ins> return result } <del> // hcsStatsToInterfaceStats converts hcsshim.NetworkStats to statsapi.InterfaceStats. func hcsStatsToInterfaceStats(stat hcsshim.NetworkStats) (*statsapi.InterfaceStats, error) { endpoint, err := hcsshim.GetHNSEndpointByID(stat.EndpointId) if err != nil { return nil, err </del> <ins> func hcsStatToInterfaceStat(hcsStats *hcsshim.HNSEndpointStats, endpointName string) statsapi.InterfaceStats { iStat := statsapi.InterfaceStats{ Name:    endpointName, RxBytes: &hcsStats.BytesReceived, TxBytes: &hcsStats.BytesSent, </ins> } <ins> return iStat } </ins> <del> return &statsapi.InterfaceStats{ Name:    endpoint.Name, RxBytes: &stat.BytesReceived, TxBytes: &stat.BytesSent, }, nil </del> <ins> // newNetworkStatsProvider uses the real windows hcsshim if not provided otherwise if the interface is provided // by the cristatsprovider in testing scenarios it uses that one func newNetworkStatsProvider(p *criStatsProvider) windowsNetworkStatsProvider { var statsProvider windowsNetworkStatsProvider if p.windowsNetworkStatsProvider == nil { statsProvider = networkStats{} } else { statsProvider = p.windowsNetworkStatsProvider.(windowsNetworkStatsProvider) } return statsProvider </ins> }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When curling the endpoint for stats we don't see network stats. Network stats to be included when calling the summary. This works with dockershim for windows. This happens because Windows containerd uses hcsv2 to create containers and we query for the networks stats using hcsv1: This will be address eventually with move to CRI only metrics KEP: but is still several releases away. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: if stats.Cpu.UsageCoreNanoSeconds != nil { result.CPU.UsageCoreNanoSeconds = &stats.Cpu.UsageCoreNanoSeconds.Value } <ins> usageNanoCores := p.getContainerUsageNanoCores(stats) if usageNanoCores != nil { result.CPU.UsageNanoCores = usageNanoCores } </ins> } if stats.Memory != nil { result.Memory.Time = metav1.NewTime(time.Unix(0, stats.Memory.Timestamp))

Related: On Windows, kubelet reports only the cumulative for cpu usage. On Linux, kubelet reports both and the instantaneous . Although In the future, most components/monitoring pipelines will switch to using the cumulative metrics, a few features still depend on the instantaneous metrics today. Examples are: , HPA/VPA. Kubelet should report this metrics on Windows to support the existing features. /cc who explained to me what features depend on the cpu metrics. /cc /cc
- is this something that's clear enough in the CRI layer that you can implement it, or do we need to fix that first?
/assign
do you have any notes or a doc you can point to that better describes what features depend on which metrics? It would be good for us to share and see if anything other than is broken
The horizontal and vertical pod autoscalers also depend on the
Documentation on metrics API and metrics server (i.e., the aggregator): The metrics-server pod keeps spewing errors like this: Running : I haven't verified whether pod autoscaling works in my cluster, but I assume they won't since metrics server seems to discard the metrics from the windows pods. I believe the implementation is in kubelet (for the docker integration), so this should not need CRI changes.
I think this is generally an issue for non-Docker CRIs (not necessarily Windows-exclusive). The kubelet source code suggests that these metrics will be gathered from the CRI (if the CRI is not Docker). The struct that kubelet expects returned from the ContainerStats RPC does not include usageNanoCores. Does this mean that, until this is resolved, custom CRIs are unable to satisfactorily report metrics to the 'metrics-server'?
- are you able to work on this?
that is a good point. For CRI runtimes, the kubelet should calculate the average over 10s to produce this metric.
Good point. We designed CRI around core metrics API, but some CRI runtimes have integrated with cadvisor directly to fill the gap while we wait for core metrics API. I think it's reasonable for kubelet to do this on behalf of all CRI runtimes. I spoke to offline, and this would probably require no more than 2 days of work. If sig-windows (e.g., can pick this up, that'd be great.

has agreed to review the changes (thanks!) If not, we should clearly document that this is not supported in Windows GA for 1.14.
thanks for the update. we will wait to hear back from
Currently, is got from docker's stats API on Windows. From docker stats API , only is available, hence is not set on Windows: Do you know any way to get instantaneous for Windows containers?
I have no idea. any idea?
I think this is broader than just a Windows issue. Officially supported tools such as the kubernetes metrics-server expect the field, but custom CRIs that conform to the interfaces defined in the package are not given the liberty to return this value.
suggested previously to cache the stats temporarily and compute the stats using the diff. David, could you chime in on how this could be done? Yes, this was mentioned in the previous comments too. There are plans to transition those components off this field, but it's WIP. We are looking for a workaround in kubelet for now.
my biggest worry is that if we say this impacts the horizonal POD autoscaler, how is that passing in our end2end and conformance tests?
Do you test HPA? It's not part of conformance, and I doubt this is included in windows testing.
As to how this could be done, in and we could just add a in the struct, which we assign the previous CpuUsage metric for each container. Then, when we need to calculate the rate, we pass the previous CpuUsage along with the current one to , and calculate a rate based on the timestamp and values.
ListPodCPUAndMemoryStats() we could just add a map[string]*runtimeapi.CpuUsage in the criStatsProvider struct I've done some tinkering on this myself and I originally thought that it would be as simple as that. It gets much more complicated when you have to work in between the existing cAdvisor-based logic. I assume the intent is to cut it out entirely eventually, but the information passed through the CRI doesn't even come close to being so comprehensive. So far I've found that this approach fixes the problem for the "/stats/summary" api, but not necessarily for the other apis. In particular, "/stats/container" seems to be completely based on cAdvisor, being unimpacted by the changes we made to those two functions.

Some monitoring tools and metrics scrapers rely on the "/stats/container" api for their metrics, which means it is insufficient to update one api without doing the other.
We don't need to fill in everything in the summary API, just "usageNanoCores". The "/stats/summary" api is the one used by the metrics server, which is why we care about it in this case. The "/stats/container" isn't something we need to implement to support this.
we a couple of tickets to validate if HPA for Windows containers works. My guess is that it works for memory and not for CPU metrics because of this issue. If this is complicated to fix, i wonder if waiting for the CRI containerD changes in windows would be the best option for supporting this.
containerd and cri-o bypass this issue by leveraging cadvisor as a temporary solution. I don't think that'd work for windows.
Sorry the for late response. HPA is actually working for Windows containers. e.g. in my cluster (v1.12.5), cpu-based HPA is working as expected: Which version of kubernetes are you referring for HPA issues?
are you running heapster, or the metrics server to serve HPA? Heapster uses the cumulative metric, but the metrics-server uses the 10 average rate metric.
metrics-server is used in my cluster.
See .
/milestone v1.14


Question: f := NewFramework("horizontal-pod-autoscaling") // CPU tests <del> It("[Skipped] should scale from 1 pod to 3 pods and from 3 to 5 (scale resource: CPU)", func() { rc = NewDynamicResourceConsumer("rc", 1, 250, 0, 400, 100, f) </del> <ins> It("[Skipped][Autoscaling Suite] should scale from 1 pod to 3 pods and from 3 to 5 (scale resource: CPU)", func() { rc = NewDynamicResourceConsumer("rc", 1, 250, 0, 500, 100, f) </ins> defer rc.CleanUp() <del> createCPUHorizontalPodAutoscaler(rc, "0.1") </del> <ins> createCPUHorizontalPodAutoscaler(rc, 20) </ins> rc.WaitForReplicas(3) rc.ConsumeCPU(700) rc.WaitForReplicas(5) }) <del> It("[Skipped] should scale from 5 pods to 3 pods and from 3 to 1 (scale resource: CPU)", func() { rc = NewDynamicResourceConsumer("rc", 5, 700, 0, 200, 100, f) </del> <ins> It("[Skipped][Autoscaling Suite] should scale from 5 pods to 3 pods and from 3 to 1 (scale resource: CPU)", func() { rc = NewDynamicResourceConsumer("rc", 5, 400, 0, 500, 100, f) </ins> defer rc.CleanUp() <del> createCPUHorizontalPodAutoscaler(rc, "0.3") </del> <ins> createCPUHorizontalPodAutoscaler(rc, 30) </ins> rc.WaitForReplicas(3) rc.ConsumeCPU(100) rc.WaitForReplicas(1) }) <del> // Memory tests It("[Skipped] should scale from 1 pod to 3 pods and from 3 to 5 (scale resource: Memory)", func() { rc = NewDynamicResourceConsumer("rc", 1, 0, 2200, 100, 2500, f) defer rc.CleanUp() createMemoryHorizontalPodAutoscaler(rc, "1000") rc.WaitForReplicas(3) rc.ConsumeMem(4200) rc.WaitForReplicas(5) }) It("[Skipped] should scale from 5 pods to 3 pods and from 3 to 1 (scale resource: Memory)", func() { rc = NewDynamicResourceConsumer("rc", 5, 0, 2200, 100, 2500, f) defer rc.CleanUp() createMemoryHorizontalPodAutoscaler(rc, "1000") rc.WaitForReplicas(3) rc.ConsumeMem(100) rc.WaitForReplicas(1) }) // Backup tests, currently disabled It("[Skipped][Horizontal pod autoscaling Suite] should scale from 1 pod to 3 pods (scale resource: CPU)", func() { rc = NewDynamicResourceConsumer("rc", 1, 700, 0, 800, 100, f) defer rc.CleanUp() createCPUHorizontalPodAutoscaler(rc, "0.3") rc.WaitForReplicas(3) }) It("[Skipped][Horizontal pod autoscaling Suite] should scale from 3 pods to 1 pod (scale resource: CPU)", func() { rc = NewDynamicResourceConsumer("rc", 3, 0, 0, 100, 100, f) defer rc.CleanUp() createCPUHorizontalPodAutoscaler(rc, "0.7") rc.WaitForReplicas(1) }) It("[Skipped][Horizontal pod autoscaling Suite] should scale from 1 pod to maximum 5 pods (scale resource: CPU)", func() { rc = NewDynamicResourceConsumer("rc", 1, 700, 0, 800, 100, f) defer rc.CleanUp() createCPUHorizontalPodAutoscaler(rc, "0.1") rc.WaitForReplicas(5) }) It("[Skipped][Horizontal pod autoscaling Suite] should scale from 1 pod to 3 pods and from 3 to 1 (scale resource: CPU)", func() { rc = NewDynamicResourceConsumer("rc", 1, 700, 0, 800, 100, f) defer rc.CleanUp() createCPUHorizontalPodAutoscaler(rc, "0.3") rc.WaitForReplicas(3) rc.ConsumeCPU(300) rc.WaitForReplicas(1) }) It("[Skipped][Horizontal pod autoscaling Suite] should scale from 3 pods to 1 pod and from 1 to 3 (scale resource: CPU)", func() { rc = NewDynamicResourceConsumer("rc", 3, 0, 0, 800, 100, f) defer rc.CleanUp() createCPUHorizontalPodAutoscaler(rc, "0.3") rc.WaitForReplicas(1) rc.ConsumeCPU(700) rc.WaitForReplicas(3) }) It("[Skipped][Horizontal pod autoscaling Suite] should scale from 1 pod to 3 pods (scale resource: Memory)", func() { rc = NewDynamicResourceConsumer("rc", 1, 0, 800, 100, 900, f) defer rc.CleanUp() createMemoryHorizontalPodAutoscaler(rc, "300") rc.WaitForReplicas(3) }) It("[Skipped][Horizontal pod autoscaling Suite] should scale from 3 pods to 1 pod (scale resource: Memory)", func() { rc = NewDynamicResourceConsumer("rc", 3, 0, 0, 100, 100, f) defer rc.CleanUp() createMemoryHorizontalPodAutoscaler(rc, "700") rc.WaitForReplicas(1) }) It("[Skipped][Horizontal pod autoscaling Suite] should scale from 1 pod to 3 pods and from 3 to 1 (scale resource: Memory)", func() { rc = NewDynamicResourceConsumer("rc", 1, 0, 700, 100, 800, f) defer rc.CleanUp() createMemoryHorizontalPodAutoscaler(rc, "300") rc.WaitForReplicas(3) rc.ConsumeMem(100) rc.WaitForReplicas(1) }) It("[Skipped][Horizontal pod autoscaling Suite] should scale from 3 pods to 1 pod and from 1 to 3 (scale resource: Memory)", func() { rc = NewDynamicResourceConsumer("rc", 3, 0, 0, 100, 800, f) defer rc.CleanUp() createMemoryHorizontalPodAutoscaler(rc, "300") rc.WaitForReplicas(1) rc.ConsumeMem(700) rc.WaitForReplicas(3) }) It("[Skipped][Horizontal pod autoscaling Suite] should scale from 1 pod to maximum 5 pods (scale resource: Memory)", func() { rc = NewDynamicResourceConsumer("rc", 1, 0, 700, 100, 800, f) defer rc.CleanUp() createMemoryHorizontalPodAutoscaler(rc, "100") rc.WaitForReplicas(5) }) </del> }) <del> func createCPUHorizontalPodAutoscaler(rc *ResourceConsumer, cpu string) { glog.Fatal("createCPUHorizontalPodAutoscaler not implemented!") // TODO: reimplemente e2e tests for the new API. } // argument memory is in megabytes func createMemoryHorizontalPodAutoscaler(rc *ResourceConsumer, memory string) { glog.Fatal("createMemoryHorizontalPodAutoscaler not implemented!") // TODO: reimplemente e2e tests for the new API. </del> <ins> func createCPUHorizontalPodAutoscaler(rc *ResourceConsumer, cpu int) { minReplicas := 1 hpa := &extensions.HorizontalPodAutoscaler{ ObjectMeta: api.ObjectMeta{ Name:      rc.name, Namespace: rc.framework.Namespace.Name, }, Spec: extensions.HorizontalPodAutoscalerSpec{ ScaleRef: extensions.SubresourceReference{ Kind:        kind, Name:        rc.name, Namespace:   rc.framework.Namespace.Name, Subresource: subresource, }, MinReplicas:    &minReplicas, MaxReplicas:    5, CPUUtilization: &extensions.CPUTargetUtilization{TargetPercentage: cpu}, }, } _, errHPA := rc.framework.Client.Extensions().HorizontalPodAutoscalers(rc.framework.Namespace.Name).Create(hpa) expectNoError(errHPA) </ins> }

e.g. scale up if CPU usage exceeds 70% of CPU request. Subtasks: [x] Add API changes [x] Add implementation [x] Add support in kubectl to print status with cpu request utilization [x] Add e2e tests
I've sent main PR for review (). During your day/my night please look into kubectl changes and e2e tests.


Question: <ins> monit: pkg: - installed /etc/monit/conf.d/etcd: file: - managed - source: salt://monit/etcd - user: root - group: root - mode: 644 monit-service: service: - running - name: monit - watch: - pkg: monit - file: /etc/monit/conf.d/etcd </ins>

We don't currently automatically restart etcd. We should.
Can we run etcd as a Pod without resource limit, so that we could benefit directly from 1) RestartPolicy: Always 2) resource monitoring and logging
When I bring up an e2e cluster (i.e. go run --up), without even running any of the e2e tests against it, it appears that monit is repeatedly killing and restarting etcd: davidopp more [UTC Feb 19 07:39:16] info : monit: generated unique Monit id and stored to '/var/lib/monit/id' [UTC Feb 19 07:39:16] info : Starting monit daemon [UTC Feb 19 07:39:16] error : State file '/var/lib/monit/state': Unable to read magic [UTC Feb 19 07:39:16] info : 'systeme2e-test-davidopp-master.c.striped-torus-760.internal' Monit started [UTC Feb 19 07:39:18] info : monit daemon with pid [5620] killed [UTC Feb 19 07:39:18] info : 'systeme2e-test-davidopp-master.c.striped-torus-760.internal' Monit stopped [UTC Feb 19 07:39:18] info : Starting monit daemon [UTC Feb 19 07:39:18] info : 'system_e2e-test-davidopp-master.c.striped-torus-760.internal' Monit started [UTC Feb 19 07:39:18] error : 'etcd' failed, cannot open a connection to INET[127.0.0.1:4001/v2/keys/] via TCP [UTC Feb 19 07:39:18] info : 'etcd' trying to restart [UTC Feb 19 07:39:18] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 07:39:18] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 07:41:18] error : 'etcd' failed, cannot open a connection to INET[127.0.0.

1:4001/v2/keys/] via TCP [UTC Feb 19 07:41:18] info : 'etcd' trying to restart [UTC Feb 19 07:41:18] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 07:41:18] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 07:43:18] error : 'etcd' failed, cannot open a connection to INET[127.0.0.1:4001/v2/keys/] via TCP [UTC Feb 19 07:43:18] info : 'etcd' trying to restart [UTC Feb 19 07:43:18] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 07:43:18] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 07:45:18] error : 'etcd' failed, cannot open a connection to INET[127.0.0.1:4001/v2/keys/] via TCP [UTC Feb 19 07:45:18] info : 'etcd' trying to restart [UTC Feb 19 07:45:18] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 07:45:18] info : 'etcd' start: /etc/init.d/etcd [.] I examined the etcd logs and there is nothing suspicious there. I suspect this may be the cause of some of the e2e test flakiness.
I think this particular problem is caused by firewall settings.
actually didn't fix the problem. Here's the [UTC Feb 19 23:22:25] info : monit: generated unique Monit id and stored to '/var/lib/monit/id' [UTC Feb 19 23:22:25] info : Starting monit daemon [UTC Feb 19 23:22:25] error : State file '/var/lib/monit/state': Unable to read magic [UTC Feb 19 23:22:25] info : 'systeme2e-test-davidopp-master.c.striped-torus-760.

internal' Monit started [UTC Feb 19 23:22:26] info : monit daemon with pid [5481] killed [UTC Feb 19 23:22:26] info : 'systeme2e-test-davidopp-master.c.striped-torus-760.internal' Monit stopped [UTC Feb 19 23:22:27] info : Starting monit daemon [UTC Feb 19 23:22:27] info : 'system_e2e-test-davidopp-master.c.striped-torus-760.internal' Monit started [UTC Feb 19 23:22:32] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:22:32] info : 'etcd' trying to restart [UTC Feb 19 23:22:32] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:22:32] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 23:24:37] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:24:37] info : 'etcd' trying to restart [UTC Feb 19 23:24:37] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:24:37] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 23:26:43] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:26:43] info : 'etcd' trying to restart [UTC Feb 19 23:26:43] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:26:43] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 23:28:48] error : 'etcd' failed,

cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:28:48] info : 'etcd' trying to restart [UTC Feb 19 23:28:48] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:28:48] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 23:30:53] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:30:53] info : 'etcd' trying to restart [UTC Feb 19 23:30:53] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:30:53] info : 'etcd' start: /etc/init.d/etcd
The reason the fix didn't work is that it hard-codes the hostname as kubernetes-master but the hostname may actually be anything. For example, in e2e test the master hostname is e2e-test-${USER}-master
This is fixed by changing etcd to startup with -bind-addr 0.0.0.0:4001 I'm sending a PR now.
(And thanks to for figuring out the problem was that etcd wasn't listening on the localhost address.)


Question: return nil } <del> name := s.loadBalancerName(service) </del> <ins> name := cloudprovider.GetLoadBalancerName(service) </ins> err := s.balancer.UpdateTCPLoadBalancer(name, s.zone.Region, hosts) if err == nil { return nil

Since this PR got merged, the ELB e2e test, which was consistently succeeding, is now consistently failing: Revert now that is in, and fix the tests that were relying on it. (details) Commit by arob Revert now that is in, and fix the tests that were relying on it. The file was modified The file was modified The file was modified I'm going to roll it back to confirm.


Question: } if o.Minify { <ins> if len(o.Context) > 0 { config.CurrentContext = o.Context } </ins> if err := clientcmdapi.MinifyConfig(config); err != nil { return err }

/kind bug What happened: Current context was set to cluster1. Ran and got back information about cluster1 instead. What you expected to happen: I expected the output to be the cluster2 information How to reproduce it (as minimally and precisely as possible): Have two contexts in your kube config, A and B Run to set A as current context Run Anything else we need to know?: I'd like to help contribute, so if this is an easy fix, someone point me in the right direction and I'll try to get it done. Environment: Kubernetes version (use ): 1.10.2 OS (e.g. from /etc/os-release): MacOS 10.11.6 Install tools: kubectl was installed via brew
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Filed


Question: if err != nil { glog.Errorf("Unable to get pod with name %q and uid %q info, health checks may be invalid", podFullName, uid) } <del> netInfo, found := podStatus.Info[dockertools.PodInfraContainerName] if found { podStatus.PodIP = netInfo.PodIP } </del> for _, container := range pod.Spec.Containers { expectedHash := dockertools.HashContainer(&container)

Kubelet now only reports ContainerStatus. PodStatus / PodCondition is observed at master side based on a set of ContainerStatus. There is no other way to get PodIP except net container for master side. This is very confusion to the end user, , , ... Now we are finalizing on v1beta3 API, including PodStatus & ContainerStatus. We can make net container completely invisible to the end users by finishing the following steps: 1) Expose network related errors at API level 2) Generate PodStaus directly from Kubelet
3) once PodStatus generated by kubelet, PodIP can be removed from ContainerStatus.
I think reporting PodStatus rather than just container status is most important. How hard is that?
Used to be hard since there are many dependencies when I first report Status back. Now it is totally feasible. I already started to write PRs for this. :-)
One more PR required to close this one.


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/runtime/serializer" </ins> "k8s.io/apimachinery/pkg/runtime/serializer/streaming" "k8s.io/apimachinery/pkg/util/diff" "k8s.io/apimachinery/pkg/util/httpstream" "k8s.io/apimachinery/pkg/util/intstr" <ins> utilnet "k8s.io/apimachinery/pkg/util/net" </ins> "k8s.io/apimachinery/pkg/watch" "k8s.io/client-go/kubernetes/scheme" restclientwatch "k8s.io/client-go/rest/watch"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !

transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.


Question: Spec: corev1.NodeSpec{ Unschedulable: true, }, <ins> Status: corev1.NodeStatus{ Capacity:    nodeCapacity, Allocatable: nodeAllocatable, }, </ins> }, &coordinationv1.Lease{ ObjectMeta: metav1.ObjectMeta{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: scheduler doesn't assign pod because 'not enough resource: hugepages-1Gi',but the node info show it has enough resource. describe node What you expected to happen: the pod was in normal status like pending and running How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It was change default hugepage size to 1Gi and hugepages=36 numastat was show there has enough hugepage also proc/meminfo was show there has enough resource other info Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
cc do you have thoughts here? I see huge pages and think of you :)
Hmm. Are you sure that there are no other pods using huge page memory ? To me, it looks like the node has (huge page @ 1GiB) memory, you are requsting , but is already used. Can you verify that by listing the pods on the node, and check if some of them are using huge page memory already?
Also, the OS will report that no huge pages are in use as long as no processes have started using the hugepages. Eg. when a pod requesting with is assigned to a node, k8s will then mark the s with huge page memory it as "used". The cgroup controller (if hugetlb cgroup is mounted) will then set the usage limit to . Before the pod actually starts using the huge page memory, the OS will report that no huge page memory is used. When a process inside the pod maps with huge page memory via eg. , the OS will see that the memory is in used. If the process tries to map more than the requested amount, the map will fail (given that the hugetlb cgroup is mounted). To verify if the hugetlb cgroup is mounted, run .

If mounted, it will retrun something like this:
I’m so sorry that there actually has a other pod using hugepagei only see node info available hugepage. so i close the issuethanks a lot
No problem I think this issue is a symptom that something should be done to make it easier to understand. I will look into it.