Question: // needsCleanup checks if load balancer needs to be cleaned up as indicated by finalizer. func needsCleanup(service *v1.Service) bool { <del> return service.ObjectMeta.DeletionTimestamp != nil && servicehelper.HasLBFinalizer(service) </del> <ins> if !servicehelper.HasLBFinalizer(service) { return false } if service.ObjectMeta.DeletionTimestamp != nil { return true } // Service doesn't want loadBalancer but owns loadBalancer finalizer also need to be cleaned up. if service.Spec.Type != v1.ServiceTypeLoadBalancer { return true } return false </ins> } // needsUpdate checks if load balancer needs to be updated due to change in attributes.

What happened: Service Load balancer may not be cleaned up on type change if below sequence happen: type changes from to controller restarts/crashes before it gets a chance to process the update from 1. controller starts up again and sees the service doesn't want load balancer, hence skips it. load balancer resource sticks around even if it is no longer needed by the service. Note that this is a different issue than , as load balancer resource will still be cleaned up when the service object is deleted, just not when the type change happens. Anything else we need to know?: This is a corner case that is not covered by the initial service load balancer finalizer implementation (). This can probably be fixed in a trivial way --- make sure the handler takes in service with non-LoadBalancer type but has finalizer attached: I will put more thoughts later. cc
/sig network
Can i reserve this issue in advance? :) I'd like to take this issue as my first issue.
Sure thing, feel free to work on this and glad that you can help :)
Do you mean attach finalizer for service that non-LoadBalancer type in AddFunc? I'm new to kubernetes, hope you can guide me. :)
No worries. I actually meant should include service with non-LoadBalancer type && with finalizer attached. As we only attach finalizer to a service only when we are creating a load balancer for it, a finalizer presents on the service (no matter what type it has) indicates there is a load balancer associated. Hence it makes sense that we should clean up the load balancer if the service no longer wants one, even if the service has not been deleted.
To clarify further, "attach finalizer for service that non-LoadBalancer type in AddFunc" would not be what we want --- we shouldn't attach finalizer to service that we will not be creating load balancer for.
After go through service controller part code, i guess you mean add a condition for as follows: . Am i right? I'm not sure when service type changes if can be involved. Can you clarify this ?
if ok && (wantsLoadBalancer(svc) (svc.Spec.Type != v1.ServiceTypeLoadBalancer && servicehelper.HasLBFinalizer(svc))) . Yea this was what I meant.

On service type change, will be involved instead of . That part is already handled. This issue is explicit for the case when controller fails to handle the type change update before it exiting (e.g. crashing, restart, etc.). Putting such detection to is more off a remedy, it means the next time service controller starts up, it can re-syncs all existing services and find out it still need to cleanup load balancer for this service, even if it doesn't have load balancer type anymore.
Thanks . I have sent a PR assined to you. I think it should be checked in function . It is the case that need to clean up whatever the reason is.


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package fake import ( "context" v1 "k8s.io/api/core/v1" policy "k8s.io/api/policy/v1" meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "testing" ) func TestNewSimpleClientset(t *testing.T) { client := NewSimpleClientset() client.CoreV1().Pods("default").Create(context.Background(), &v1.Pod{ ObjectMeta: meta_v1.ObjectMeta{ Name:      "pod-1", Namespace: "default", }, }, meta_v1.CreateOptions{}) client.CoreV1().Pods("default").Create(context.Background(), &v1.Pod{ ObjectMeta: meta_v1.ObjectMeta{ Name:      "pod-2", Namespace: "default", }, }, meta_v1.CreateOptions{}) err := client.CoreV1().Pods("default").EvictV1(context.Background(), &policy.Eviction{ ObjectMeta: meta_v1.ObjectMeta{ Name: "pod-2", }, }) if err != nil { t.Errorf("TestNewSimpleClientset() res = %v", err.Error()) } pods, err := client.CoreV1().Pods("default").List(context.Background(), meta_v1.ListOptions{}) // err: item[0]: can't assign or convert v1beta1.Eviction into v1.Pod if err != nil { t.Errorf("TestNewSimpleClientset() res = %v", err.Error()) } else { t.Logf("TestNewSimpleClientset() res = %v", pods) } } </ins>

After I evicted an pod, I performed the operation of the full list instance, but found something that did not meet expectations After pods are evicted, non-evicted ones can be listed normally No response <details</detailsv1.20.0 <details</details<details</details<details</details<details</details<details</details
/sig node
/sig api-machinery /remove-sig node
/remove-sig api-machinery
/assign since you are tagged on the PR. Thanks! /triage accepted
/sig testing
If you have time, please help to look at this pr, thanks
this is a general issue with subresources that accept create requests of different object types than the parent resource see also:


Question: package rbd import ( <ins> "encoding/json" </ins> "errors" "fmt" "math/rand"

etcd cluster backing apiserver died spontaneously, after restart apiserver pulled up the pods on different machines. Node storage was backed by ext4 over rbd, which became mounted on several nodes at once. That could have possibly led to data corruption. I guess that the proper way to mount rbd volume is to take a lock on it first, which will at least fail if it's locked somewhere else, requiring manual intervention, but not destroying data. The lock should be optional, in case a proper cluster fs is mounted over the rbd. In this case there was clearly no need for accessing the data from several pods so ext4 is deemed a proper fs.
, Vladimir Pouzanov < :
want to get some detail. How is your pod-to-rbd mapping like? Since rbd image is a block device, it can be used by one pod at a time. I am not sure if locking is a proper way to go - if the pod holds a lock but dies in the middle of mounting, the rbd image (or any other volume type) is then excluded from access. In such case, a high level coordination at apiserver is a better place to resolve this problem.
Here's my rc: I guess it's better to have a lock to be cleaned up manually rather than to have non-cluster fs to be mounted in several locations at once?
thanks, this pod looks good to me. what would cause multiple nodes run the same pod?
I don't know the exact reason of why kubelet didn't unmount the old fs, but after apiserver and scheduler were restarted pod got allocated on different node.
It is a scary thing to see etcd down, it is meant to be reliable...
Unfortunately I don't have long-term logging in test cluster so I have no idea why it actually died. but using kubedns as primary node resolver was clearly a bad idea, once it died, nodes stuck with no resolver at all.
your thought on HA?
Wrong Tim St. Clair?
- you HAVE to be ready to handle network partitions - the old node might still be alive and even using the volume, but the apiserver can't see the node, so it thinks it deletes the pod and starts it elsewhere. If RBD corrupts itslef in this case, it is not useful and we should strip it from the codebase.

, Huamin Chen wrote:
that makes sense. rbd doesn't corrupt itself. This is inconsistency happens when you have two pods mounting the same rbd image. How does GCE PD handle/prevent this case?
It doesn't allow more than one read-write attachment. Multiple read-only attachments is OK. Better to fail than to allow corruption. , Huamin Chen wrote:
That's neat. I have a lock fencing for rbd, testing now.


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package options import ( "context" "fmt" "os" "path/filepath" "testing" "github.com/stretchr/testify/assert" "k8s.io/klog/v2" "k8s.io/kubernetes/pkg/scheduler/apis/config" ) const ( apiVersionMissing = "'apiVersion' is missing" apiVersionTooOld  = "no kind "KubeSchedulerConfiguration" is registered for" + " version "kubescheduler.config.k8s.io/v1alpha1"" fileNotFound = "no such file or directory" // schedulerConfigMinimalCorrect is the minimal // correct scheduler config schedulerConfigMinimalCorrect = ` apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration` // schedulerConfigDecodeErr is the scheduler config // which throws decoding error when we try to load it schedulerConfigDecodeErr = ` kind: KubeSchedulerConfiguration` // schedulerConfigVersionTooOld is the scheduler config // which throws error because the config version 'v1alpha1' // is too old schedulerConfigVersionTooOld = ` apiVersion: kubescheduler.config.k8s.io/v1alpha1 kind: KubeSchedulerConfiguration ` ) func TestLoadConfigFromFile(t *testing.T) { tmpDir, err := os.MkdirTemp("", "scheduler-configs") if err != nil { t.Fatal(err) } defer os.RemoveAll(tmpDir) correctConfigFile := filepath.Join(tmpDir, "correct_config.yaml") if err := os.WriteFile(correctConfigFile, []byte(schedulerConfigMinimalCorrect), os.FileMode(0600)); err != nil { t.Fatal(err) } decodeErrConfigFile := filepath.Join(tmpDir, "decode_err_no_version_config.yaml") if err := os.WriteFile(decodeErrConfigFile, []byte(schedulerConfigDecodeErr), os.FileMode(0600)); err != nil { t.Fatal(err) } versionTooOldConfigFile := filepath.Join(tmpDir, "version_too_old_config.yaml") if err := os.WriteFile(versionTooOldConfigFile, []byte(schedulerConfigVersionTooOld), os.FileMode(0600)); err != nil { t.Fatal(err) } tests := []struct { name           string path           string expectedErr    error expectedConfig *config.KubeSchedulerConfiguration }{ { name:           "Empty scheduler config file path", path:           "", expectedErr:    fmt.Errorf(fileNotFound), expectedConfig: nil, }, { name:           "Correct scheduler config", path:           correctConfigFile, expectedErr:    nil, expectedConfig: &config.KubeSchedulerConfiguration{}, }, { name:           "Scheduler config with decode error", path:           decodeErrConfigFile, expectedErr:    fmt.Errorf(apiVersionMissing), expectedConfig: nil, }, { name:           "Scheduler config version too old", path:           versionTooOldConfigFile, expectedErr:    fmt.Errorf(apiVersionTooOld), expectedConfig: nil, }, } logger := klog.FromContext(context.Background()) for i, test := range tests { t.Run(fmt.Sprintf("case_%d: %s", i, test.name), func(t *testing.T) { cfg, err := LoadConfigFromFile(logger, test.path) if test.expectedConfig == nil { assert.Nil(t, cfg) } else { assert.NotNil(t, cfg) } if test.expectedErr == nil { assert.NoError(t, err) } else { assert.ErrorContains(t, err, test.expectedErr.Error()) } }) } } </ins>

I want to be exported as a public function imports scheduler code and runs scheduler simulations for pods to decide which nodegroup should be scaled up. consider this case: a user is running a scheduler with custom scheduler config the user wants cluster-autoscaler to run scheduler simulations based on the custom scheduler config For ii default scheduler config is used in cluster-autoscaler scheduler simulations. Right now, it is not possible to use custom scheduler config for cluster-autoscaler scheduler simulations but I am working on a PR () to support this feature. For this feature, I need to parse scheduler config. . It would help if we can have exported as (just like ) so that programs which need to parse scheduler config can use the exported function directly instead of writing custom code.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
PR:
/assign vadasambar
wdyt?
/sig autoscaling
LGTM, thanks.


Question: // Verify host, _, _ := net.SplitHostPort(dialAddr) <ins> if tlsConfig != nil && len(tlsConfig.ServerName) > 0 { host = tlsConfig.ServerName } </ins> if err := tlsConn.VerifyHostname(host); err != nil { tlsConn.Close() return nil, err

an APIService that routes to an external API server (e.g. service catalog) via a service in namespace a caBundle in the APIService the external API server with a certificate signed for Result: Normal API requests (get/put/post) work as expected Normal watch requests work as expected Upgrade requests (e.g. websocket watch requests) return a 500 error with text like cc


Question: var internalLabelKeys []string = []string{containerTypeLabelKey, containerLogPathLabelKey, sandboxIDLabelKey} // NOTE: Anything passed to DockerService should be eventually handled in another way when we switch to running the shim as a different process. <del> func NewDockerService(client dockertools.DockerInterface, seccompProfileRoot string, podSandboxImage string, streamingConfig *streaming.Config, pluginSettings *NetworkPluginSettings, cgroupsName string) (DockerService, error) { </del> <ins> func NewDockerService(client dockertools.DockerInterface, seccompProfileRoot string, podSandboxImage string, streamingConfig *streaming.Config, pluginSettings *NetworkPluginSettings, cgroupsName string, kubeCgroupDriver string) (DockerService, error) { </ins> c := dockertools.NewInstrumentedDockerInterface(client) ds := &dockerService{ seccompProfileRoot: seccompProfileRoot,

Kubernetes version (use ): v1.5.0-alpha.1.2189+-dirty docker: 1.12.3 Environment: fedora24 docker-engine-1.12.3-1.fc24.x86_64 What happened: if docker Cgroup Driver=cgroupfs, and set --cgroup-driver=systemd in kubelet, then enable --experimental-cgroups-per-qos=true; The container cgroup some is not correct. How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: 1) docker Cgroup Driver=cgroupfs and --cgroup-driver=cgroupfs in kubelet works well 2) docker Cgroup Driver=systemd and --cgroup-driver=systemd in kubelet works well
help check, thanks.
In Kubernetes 1.6, we can work to detect this case and return an error. IMHO its a configuration error to have docker running with a cgroupfs driver paired with the kubelet running the systemd cgroup driver. /cc
We can also document the compatibility matrix for now.


Question: }, expected: []string{ `metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead`, <del> `metadata.annotations[seccomp.security.alpha.kubernetes.io/pod]: deprecated since v1.19, non-functional in a future release; use the "seccompProfile" field instead`, `metadata.annotations[container.seccomp.security.alpha.kubernetes.io/foo]: deprecated since v1.19, non-functional in a future release; use the "seccompProfile" field instead`, </del> <ins> `metadata.annotations[seccomp.security.alpha.kubernetes.io/pod]: non-functional in v1.27+; use the "seccompProfile" field instead`, `metadata.annotations[container.seccomp.security.alpha.kubernetes.io/foo]: non-functional in v1.27+; use the "seccompProfile" field instead`, </ins> `metadata.annotations[security.alpha.kubernetes.io/sysctls]: non-functional in v1.11+; use the "sysctls" field instead`, `metadata.annotations[security.alpha.kubernetes.io/unsafe-sysctls]: non-functional in v1.11+; use the "sysctls" field instead`, },

Graduate seccomp to GA, as defined on the respective . This is an umbrella issue to track all outstanding work. [x] API Server changes (MERGED) [x] Kubelet changes (MERGED) [x] Mark flag as deprecated (MERGED) [x] Update PSP to support seccomp fields (MERGED) [x] Document Kubernetes seccomp support (MERGED)[x] Add seccomp least privilege for docker sandbox (MERGED) [x] Add seccomp least privilege for kuberuntime (MERGED) ) /area kubelet /area kubelet-api /sig node /area security /assign
/cc
just a heads up that we have concluded all the core items for seccomp to go GA. There are still 2 pending PRs that ideally would also go in 1.19, but we are waiting for them to be reviewed: and
needs to be rebased apparently and isn't just waiting in the merge pool as verify is not passing.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think the warning mechanism was blocked by a not existing API. This is now tracked in
/remove-lifecycle stale
Updated the description to given that the warning mechanism is now set to that milestone.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/priority backlog
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen What is the remaining work here?
I guess we have to follow-up on and implement: I can take care of that if we agree that we want to ship that in 1.27. /cc


Question: containerEnvSecrets  []envSecrets } <del> func podWithSecrets(ns, name string, toAttach secretsToAttach) *v1.Pod { </del> <ins> func podWithSecrets(ns, podName string, toAttach secretsToAttach) *v1.Pod { </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: ns, <del> Name:      name, </del> <ins> Name:      podName, </ins> }, Spec: v1.PodSpec{}, }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Occasionally (more frequently if the node is heavily loaded) the node fails to mount a secret or configmap for a pod. Error recovery fails and the pod must be replaced with a new pod to have any hope of getting the secret or configmap volume mounted. The problem appears to be that if a pod is updated, its secrets are removed from the secretStore's secrets map. See: Normally the effects of the bug are not felt since the usual order of operations masks the bug: 1) The pod is created and registered the first time with the secrets manager 2) The volume mounter gets the secret 3) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) If the race is lost, though, then the bug symptoms are manifest: 1) The pod is created and registered the first time with the secrets manager 2) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) 3) The volume mounter tries to get the secret and fails with the message What you expected to happen: On update, the secrets manager should only delete secrets if they are not present in the latest pod definition (rather than deleting all prior secrets). Whether the pod update happens before or after the volume mounter attempts to get the secret, it should succeed. How to reproduce it (as minimally and precisely as possible): Because it is a race condition, it's difficult to reproduce. We've most often seen the bug on systems that were under load. 1) Create a pod that has a secrets or configmap volume 2) See if the pod fails to mount the volume. Anything else we need to know?: If the above analysis is correct, this bug has been present since 1.6 for secret volumes, and 1.7 for configmap volumes. Environment: We've seen the bug in software based upon the 1.6 release.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
The PR addresses this issue. /sig node
This may be a false alarm. I've never actually reproduced it in Kubernetes proper, only in OpenShift. Since our smoking gun was anything but, I'm going to close this until I manage to find an actual problem in Kubernetes.
To close the loop on this for any that find it via future searches: It turns out that this was a false alarm. The above analysis was flawed and this turns out not to be a bug (or maybe a very minor one if we consider it a bug to log a an error on a (semantic) non-error). The error messages like were not an indicator of a failure to mount a secret volume, but instead a failure to update a secret volume during pod teardown. Here's how the error message is triggered: pod is created volume manager gets the secret and mounts the volume the lifetime of the pod, the volume manager periodically re-checks the secret to make sure that it's still in a good state. pod goes away volume is unmounted If one of the refreshes from step 3 happens after step 4 but before step 5, the log message will appear.


Question: framework.RunIfContainerRuntimeIs("docker") }) <del> Context("when shared PID namespace is enabled", func() { It("processes in different containers of the same pod should be able to see each other", func() { // TODO(yguo0905): Change this test to run unless the runtime is // Docker and its version is <1.13. By("Check whether shared PID namespace is supported.") isEnabled, err := isSharedPIDNamespaceSupported() framework.ExpectNoError(err) if !isEnabled { framework.Skipf("Skipped because shared PID namespace is not supported by this docker version.") } By("Create a pod with two containers.") f.PodClient().CreateSync(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "shared-pid-ns-test-pod"}, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:    "test-container-1", Image:   "busybox", Command: []string{"/bin/top"}, }, { Name:    "test-container-2", Image:   "busybox", Command: []string{"/bin/sleep"}, Args:    []string{"10000"}, }, }, }, }) By("Check if the process in one container is visible to the process in the other.") pid1 := f.ExecCommandInContainer("shared-pid-ns-test-pod", "test-container-1", "/bin/pidof", "top") pid2 := f.ExecCommandInContainer("shared-pid-ns-test-pod", "test-container-2", "/bin/pidof", "top") if pid1 != pid2 { framework.Failf("PIDs are not the same in different containers: test-container-1=%v, test-container-2=%v", pid1, pid2) } }) }) </del> Context("when live-restore is enabled [Serial] [Slow] [Disruptive]", func() { It("containers should not be disrupted when the daemon shuts down and restarts", func() { const (

Once the API for is finalized, add a node e2e test. /kind feature /assign /cc /sig node
We plan on adding this for 1.10


Question: kube-up echo "... calling validate-cluster" >&2 <del> if [[ "${EXIT_ON_WEAK_ERROR}" == "true" ]]; then validate-cluster else validate-cluster validate_result="$?" if [[ ${validate_result} != "0" ]]; then if [[ "${validate_result}" == "1" ]]; then exit 1 elif [[ "${validate_result}" == "2" ]]; then echo "...ignoring non-fatal errors in validate-cluster" >&2 else echo "Got unknown validate result: ${validate_result}" fi </del> <ins> # Override errexit (validate-cluster) && validate_result="$?" || validate_result="$?" # We have two different failure modes from validate cluster: # - 1: fatal error - cluster won't be working correctly # - 2: weak error - something went wrong, but cluster probably will be working correctly # We always exit in case 1), but if EXIT_ON_WEAK_ERROR != true, then we don't fail on 2). if [[ "${validate_result}" == "1" ]]; then exit 1 elif [[ "${validate_result}" == "2" ]]; then if [[ "${EXIT_ON_WEAK_ERROR}" == "true" ]]; then exit 1; else echo "...ignoring non-fatal errors in validate-cluster" >&2 </ins> fi fi

Looks like this test is never able to succeed because service does not exist: /kind failing-test /sig instrumentation
possibly related to
/assign
/assign
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
where did we get with this?
Looks like those tests didn't work for at least half a year and nobody cared... Should we delete this e2e test as well? If so I can help with that.
Yes, let's remove them. Would appreciate the help! /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Still failing: /remove-lifecycle stale


Question: t.Errorf("Expect command executed %d times, but got %d", svcCount, fexec.CommandCalls) } } <ins> func TestIsIPv6String(t *testing.T) { testCases := []struct { ip         string expectIPv6 bool }{ { ip:         "127.0.0.1", expectIPv6: false, }, { ip:         "192.168.0.0", expectIPv6: false, }, { ip:         "1.2.3.4", expectIPv6: false, }, { ip:         "bad ip", expectIPv6: false, }, { ip:         "::1", expectIPv6: true, }, { ip:         "fd00::600d:f00d", expectIPv6: true, }, { ip:         "2001:db8::5", expectIPv6: true, }, } for i := range testCases { isIPv6 := IsIPv6String(testCases[i].ip) if isIPv6 != testCases[i].expectIPv6 { t.Errorf("[%d] Expect ipv6 %v, got %v", i+1, testCases[i].expectIPv6, isIPv6) } } } func TestIsIPv6(t *testing.T) { testCases := []struct { ip         net.IP expectIPv6 bool }{ { ip:         net.IPv4zero, expectIPv6: false, }, { ip:         net.IPv4bcast, expectIPv6: false, }, { ip:         net.ParseIP("127.0.0.1"), expectIPv6: false, }, { ip:         net.ParseIP("10.20.40.40"), expectIPv6: false, }, { ip:         net.ParseIP("172.17.3.0"), expectIPv6: false, }, { ip:         nil, expectIPv6: false, }, { ip:         net.IPv6loopback, expectIPv6: true, }, { ip:         net.IPv6zero, expectIPv6: true, }, { ip:         net.ParseIP("fd00::600d:f00d"), expectIPv6: true, }, { ip:         net.ParseIP("2001:db8::5"), expectIPv6: true, }, } for i := range testCases { isIPv6 := IsIPv6(testCases[i].ip) if isIPv6 != testCases[i].expectIPv6 { t.Errorf("[%d] Expect ipv6 %v, got %v", i+1, testCases[i].expectIPv6, isIPv6) } } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There is a bug in ipv6, see: What you expected to happen: The judgement logic should be: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network /area ipv6
/assign cc


Question: scaleSet    string vmList      []string nodeName    string <ins> location    string </ins> zone        string faultDomain int32 expected    string

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider gets the availability zone from IMDS directly and the zone is usually in lower cases in most regions. But there're chances this is not true in some regions, hence customer's workloads depend on availability zone may not work. What you expected to happen: The availability zone is always in lower cases. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig cloud-provider /area provider/azure
/assign
Fixed by .


Question: } if endpointPort.AppProtocol != nil { <del> for _, msg := range validation.IsQualifiedName(*endpointPort.AppProtocol) { allErrs = append(allErrs, field.Invalid(idxPath.Child("appProtocol"), endpointPort.AppProtocol, msg)) } </del> <ins> allErrs = append(allErrs, apivalidation.ValidateQualifiedName(*endpointPort.AppProtocol, idxPath.Child("appProtocol"))...) </ins> } }

After got merged, we have new helper function here: It will be good for us to use this simpler helper function to replace previous function. /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : After got merged, we have new helper function here: </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig usability
I would like to take this one /assign
I worked in on this as well, but kept your changed files untouched (as written in my PR description). same for you. I hope this is OK for you.
This should be reopened until and got merged (my PR excluded the changes from the other PRs).
/reopen per last comment
Reopened this issue. <detailsIn response to : This should be reopened until and got merged (my PR excluded the changes from the other PRs). per last comment Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
can you assign me
/assign
/assign
I'd love to work on this issue.

/assign
Hi, , I am a first commitor for kubernetes and I have committed a PR in , could you help me to review it, or give me some suggestions. Thanks a lot.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "syscall" "github.com/vishvananda/netlink" <ins> "github.com/vishvananda/netlink/nl" </ins> "github.com/appc/cni/libcni" "github.com/golang/glog"

I'm not sure where the defaults come from but they sometimes drop all packets causing container IPs to become unreachable. ARP fails. I'm not sure if this is related but I also noticed on this machine that the default pfifo qdisc drops all packets. pfifo is the default qdisc for an htb node so it's used in the unconfigured htb qdisc: This is because the default value for queue size is zero. On other machines I've seen the default value set to 1 packet and 1000 packets. When I set the limit explicitly the container becomes reachable again. The broken configuration is applied by kubelet to implement traffic shapping and there is no option to turn this off. This patch fixes the kubelet on broken machines by disabling traffic shapping: cc
does this happen with --configure-cbr0 and packet shaping too? The kubenet implementation just uses the existing kubernetes shaping code that --configure-cbr0 does.
It seems the *fifo qdiscs default their packet limit to the txquelen of the interface (man tc-pfifo). It also seems most virtual interface types (bridge, etc) set a dev-txqueuelen of 0. It further seems that relies on the default packet limit. I could swear this worked before with kubenet, and it must have worked with --configure-cbr0 when the code was originally . I'll check around to figure out what the default limit should be for interfaces that don't specify a tx queue length.
Note that to test with --configure-cbr0 you'd also need --reconcile-cidr.
Upstream CNI fix is here: and we'll also work around it in kubenet for now.
FWIW, the upstream CNI fix got merged for CNI 0.3.0


Question: // CreateDisk implements Volumes.CreateDisk func (c *Cloud) CreateDisk(volumeOptions *VolumeOptions) (KubernetesVolumeID, error) { <del> allZones, err := c.getAllZones() </del> <ins> allZones, err := c.getCandidateZonesForDynamicVolume() </ins> if err != nil { return "", fmt.Errorf("error querying for all zones: %v", err) }

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Install tools: default install What happened: AWS EBS backed PersitantVolumeClaim created a volume in an AZ where there was a master but no nodes. Making it unusable. What you expected to happen: Zones with master and no nodes shouldn't be used to create the EBS Volume How to reproduce it (as minimally and precisely as possible): AWS setup across 3 AZs in one region Master in three AZs Nodes in 2 AZs (so one AZ has a Master but no Nodes) Create a Storage class to use EBS without specifying AZCreate PVC against StorageClassPVC can create a volume in an availability zone with no nodes Anything else do we need to know: Mentioned on Slack channel #sig-aws with who asked me to raise this
, Mengqi Yu wrote:
Running Kubernetes master in a zone that has no running nodes is quite strange and I don't think this is a typical case we should support. AWS cloud provider lists all machines in the cluster and chooses a random zone that has at least one running instance. It seems it does not distinguish master/node machines. IMO, GCE has the same problem, it lists all zones in a region where the master runs and it does not distinguish masters and nodes either. We can either introduce a new AWS instance tag with master/node role that is running on the instance or we can move the zone decision into volume plugin itself, where we have a kubeclient and we can filter nodes relatively easily, however we will depend on an admission controller that must add a zone label to nodes. thoughts? As a third option, we could rely on kubernetes admin to fill all zones into StorageClass in this weird case. Right now we support only one zone in "zone" parameter, we could allow a list:
just to add to this, the reason why it's a deal is I actually don't have full control over where the nodes run, as they're in an autoscale group across all AZs. So you could have master in AZ a and nodes in b and c just because that's what the auto scale group picked.
I just ran into this, however, I only have nodes / masters in 3 zones, but the PV was created in a 4th zone.

So right now I can't even spin up a node in the last AZ since in my infrastructure, I'm not supporting it.
I think it may be more than just where there are or not nodes vs masters in an AZ. As an additional use case, consider pods that must run on a "dedicated" tenancy node in AWS, (which have a baseline cost per-zone) for compliance. Since masters are supposed to be an odd number (i.e. 3 zones) it would make sense to have 3 instead of 2. But for a pod that has to be scheduled on a "dedicated" tenancy node, there may be only 2 zones with nodes that are "dedicated" due to said cost. Even if there are nodes in all zones, a pod may need to be scheduled on a specific node or subset thereof. More information about node selection: More information about why this specific case would be necessary here: If PVs are used by pods, and pods must be able to access them from whatever nodes they are constrained/scheduled to, I think whatever solution is implemented needs to account for that.
do you have any other non-Kubernetes instances in us-east-1b in your AWS account? Did you try to tag your AWS instances that run Kubernetes with tag - all AWS instances in one Kubernetes cluster must share the same tag value, so you can run multiple Kubernetes clusters in one AWS account. This tag is not documented, see .
There is another non-k8s instance running in us-east-1b, but it's not related (created via breanstalk). I labels to all my nodes / controller as you sugessted and still it created the PV in . I may bail on Statefulset for now and manage my pv's by hand since I'm building out an operator and it wont be that painful, but would like to see if this could be addressed. I'm also willing to help out if you can help point into the right places this logic lives. I'll also try and dig in and see if I can find it myself.
It would be ideal to have the StorageClass accept an expression or wildcard zone under the aws provisioner. Ideally I shouldn't have to care where my pods are running and where the EBS are created.
We've a PR that adds "zones:" with list of zones instead of single "zone:". No wildcards though.

I just hit this as well. I think moving the "zone" decision closer to the provisioning decision would be best. Perhaps wait until a PVC is consumed by a pod before you create it in a zone. That way corner cases like this are not a problem.
Just want to check whether PR is good enough to solve the issue raised here?
, is right. Another use case is that if pod consumes 1 PVC, associated PVs might be created in different zones, therefore pod becomes unschedulable, as there is no node which belongs 1 AZ. Zone decision when provisioning PV must be made as late as possible, while currently it seems to be a very first thing to do.
I don't think it can be closed yet, reasons are: those magical tags are not documented anywhere, how people or going to know to use them? can't it continue to create volume in a zone where pod with a given PVC can never be scheduled (lets say it has node affinity configured to a node in a particular AZ)?
Would it make sense to say that the dynamic creation of a volume should happen only once a PVC has been attached to a pod which has been successfully scheduled to a node? Then the av_zone of the volume should match that of the selected node?
Yes I would agree with that statement. The PVC should be created when we know the node and AZ it is a apart of.
Just to add another use case to this - I just spun up a new cluster in AWS. Masters spanning , , and . After creating the ASG for nodes, I learned that AWS is out of instance capacity in for that particular instance type. Now I currently don't have nodes in (until capacity is restored and I can move an instance over). The reason this is different is because, in this case, I don't want to tell k8s to only use a subset of AZs. I want it to deploy across all AZs that are available. It just so happens that at this moment in time not all AZs are available. suggestion above would work, I believe. Schedule the node first, then provision a volume. That might cause other problems, though.
I just ran into this today - PVC created in eu-west-1c, but the only node with memory to to run the pod was in eu-west-1b.

This was a new pod and a new volume claim. Can we have this re-opened at least please?
We had to ultimately get rid of all persistence volume claims on AWS because of this. For really large clusters with lots of reserved capacity in all availability zones that woulnd't be such a big deal, but I'd image most companies are trying to optimize their cost and don't have that many node pre-provisioned just for the sake of it. Still looking for a solution which is easy to configure and AZ-failure friendly
Right now, there isn't really. You have a few options: EFS - it is regional, not zonal. You have to be OK working with NFS as the protocol, and potential performance implications. To be fair, NFS has gotten quote good, I know companies using it to store critical data. I am not an NFS expert, so don't ask my opinions here... Commercial solutions - Portworx and StorageOS both provide local storage, backed by local disk or EBS, but replicated at the app level (their app) between nodes, across zones, so you don't depend on zonal resources (like EBS). DIY - have fun. Ceph, some other solutions. For one use case with which I am involved, we are living with it, and using EFS where necessary. We don't love it, but the resources to implement something else aren't there now.
Would in a StorageClass go some way to fixing the issue? (At least for new claims anyway)? From Edit: I tried this, and the kube-controller-manager logs say: but the pod is FailedScheduling "2 node(s) didn't find available persistent volumes to bind.".
I solved this in the Elastic Operator by creating StorageClasses per zone and spreading work across that way. You need to make sure your app is zone tolerant. For example, the Elastic operator sets up sharding and replication across zones, given one AZ goes down, the other two still have enough of the data to continue functioning.
doesn't each pod then need access to storage from a different class (appropriate for the zone it is in)? How does that work?
Just ran into this as well, running work load specific zones in 2 regions for development to save cost over 3, had an EBS volume provisioned in the zone with no instance.

should address the above scenarios when you use set in the storageclass. The plumbing for went into EBS in 1.12 as a beta feature.
I have a similar issue: 3 masters + 3 nodes in 3 regions. I have a PV (created on regionA) for data backups created by a CronJob. When the cron's job gets scheduled on a node on a different region (regionA's CPU has been allocated almost fully and the master chooses another node in a different region) - it can't start because the PV is not accessible. Is there a way to avoid this issue?
I am also facing the same issue with 1.10 version of Kubernetes. Nodes are running in two regions, PV got created in the same region. still, I am getting
As mentioned above, this feature is now in beta in 1.12:
Are there any workaround for Amazon EKS to address the above issue? I am trying to make a multi-az JupyterHub deployment and changing zones causes the "no available volume zone" error...
Is this still in progress? I just ran in to issue as well
we moved it to GCP with 1.10 version and didn't face the issue
I'm also having this issue... So there's no solution to this?
im also having this issue on AWS EKS Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m (x841 over 1h) default-scheduler 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone. losing hope over here :(
Warning FailedScheduling 3m2s (x660 over 58m) default-scheduler 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone.
Another similar case here, like 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone. Like the use case by but the reason is not memory is pods allocation. The EBS was created in the AZ where the node is full (1c).. you can guess the rest..
Any update on this ? Since Node affinity is a beta feature causing this issue?

Edit: (Beware: Updating your EKS cluster can potentially screw things up & cause nodes and volumes in different zones)
I'm using EKS to deploy my services integrating it with rancher, I ran into the same issue as mentioned above. My setup is something like this: have few labels for the worked nodes to seperate the nodes per namespace/team as each team requires different types of instances. created the nodes for the pods in that node needs persistant volumes in a single availability zone as aws EBS attachment is per availability zone. I deployed my applications, the persistant volume got created but in different availability zone as I used the default storage class which creates EBS in random zones. The issue is fixed by creating another storage class with only single availbility zone.
I am using EKS , and I designed not only my pods to be Cattle, but also my nodes are Cattle as well : I have a Pod created by statefulset. This statefulset defines a PVC. The PVC triggers the creation of new EBS in the same AZ of the Node (i don't have pb here) Because Nodes are Cattle (destoryable any time), it happen that there is no Node exist in the AZ of EBS (PVC) =Pod of statefulset remains in "Pending" state =Pod events shows this error: " 1 node(s) had no available volume zone" Once a New node joins the cluster in the wanted AZ, the Pod of statefulset runs like a charm because the EBS can be attached at that node (same AZ). Thinking to use EFS ( )but looking for better solutions.
from my experience, EFS performance is not that great. better avoid it.
It depends how you use it. It's not great for lots of little files (don't check out a git repo on it, for instance ). It's been working great for us in production for almost a year so far, mainly on larger PDF files etc. But, like anything, definitely test it for your use case.
So I do have a question. If I just put the PVC into the specific availability. zone. Does that allow the other pods on different nodes to use it?

The nodes running on other availability zones couldn't access this PV until if you use EFS rather than EBS
I ran into a similar issue today investigating a error I have a PVC: and originally it bound fine, with a PV in us-east-1d Then, I ran an upgrade and none of the new nodes were created in us-east-1d And, now I have a disk in a zone without any nodes.
I found the issue. The issue is that when you are using AWS then the PVC is set to the Node. That means that if you try to put that pod or deployment on another node then it will not connect with the PVCs and being a pending state. It's an AWS thing. I resolve this by specifying taints and toleration to the nodes within that specific region.
thank you! can you post an example for more elaboration


Question: ], "@io_bazel_rules_go//go/platform:windows": [ "//pkg/windows/service:go_default_library", <ins> "//vendor/golang.org/x/sys/windows:go_default_library", </ins> ], "//conditions:default": [], }),

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Windows nodes running with high (90%) CPU usage are having calls to /stats/summary time out with "Context Deadline Exceeded". When this happens metrics-server cannot gather accurate CPU/memory usage for the node or pods running on the node which results in HPA not functioning returning <unknowninstead of values What you expected to happen: Windows nodes should be able to run with high CPU utilization without negatively impacting Kubernetes functionality How to reproduce it (as minimally and precisely as possible): a cluster with Windows Server 2019 nodes CPU intensive workloads into cluster ex: workload to put node into high CPU state node metrics with Anything else we need to know?: This issue appears to be easier to reproduce on 1.18.x nodes vs 1.16.x or 1.17.x nodes Ex: ! ! metrics-server logs show E1020 10:58:24. 1 ] unable to fully collect metrics: unable to fully scrape metrics from source kubeletsummary:akswin000000: unable to fetch metrics from Kubelet akswin000000 (10.240.0.115): Get https://10.240.0.115:/stats/summary?onlycpuandmemory=true: context deadline exceeded Kubelet logs show and Environment: Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: AKS / Azure OS (e.g: ): WIndows Server 2019 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: Docker EE version 19.3.11
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig windows /area kubelet
I was able to reproduce on 1.18.8 during a scale operation: where it fully subscribes all CPU's in Allocatable: Metric server logs: I do not see "context deadline" in kubelet but I did find some logs assocated: Once the pods are scheduled the node remains at 100% cpu and I no longer see errors but is not accurate: on the node (run after the pod deployment stabilized)
Adding to kubelete's flag and I no longer see the errors. This reserves some cpu for kubelet and other processes. We call this out in the but do not do the same for cpu.
Also, without setting system reserved CPU, it looks like does not accurately reflect CPU usage on the nodes when CPU utilization is very high. When I got some nodes into this state I did not see any error messages in logs from metrics-server and also I noticed that that querying the metrics API directly from showed recent timestamps.
I run into this again during scaling, collectiong logs for analysis here: Metric Server logs: Kubelet logs:
have you experienced this at all?
I believe has seen high CPU issues before. Not particularly on this. I've heard there were kernel issues in the past that triggered high CPU cases.
Some investigations on our end are showing that the DiagTrack service running in the infra/pause container can consume non-trivial amounts of CPU. I'll create an issue to address that problem as it might help out some here. That certainly will not eliminate the problem we are seeing tho.
is an issue for DiagTrack
The differences in behavior between 1.17. and 1.18. clusters looks like it is caused by (which was NOT backported to any prior releases) K8s version <= 1.17 behavior Prior to these changes CPU weights for containers were inferred based on limit values set in the pod specs. The observed behavior on Windows system when CPU weights were used was that under high-CPU load pods/containers were not guaranteed their scheduling time. The good news here is that system critical services like kubelet/docker/etc were still able to get enough CPU time for the nodes to function properly. K8s version = 1.18 behavior After these changes CPU limits were enforced for pods.

The observed behavior on 1.18 clusters is the when a Windows node enters a high-CPU state is that containers are guaranteed their scheduling time but host resources start to get starved. Usually this manifests as the node entering the state, metrics not working correctly (because calls to /stats/summary are timing out) or both. Suggestion / possible fixes Set the Windows process priority of the kubelet to  to help ensure it gets CPU time Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) Set reserve CPU with with --system-reserves=[cpu=500m] This needs to scaled based on number of CPUs on host based on my experimentations Currently guidance on for Windows only mentions reserving system-memory It is still possible to over-commit CPU and starve kubelet/docker/HCS if users schedule pods with mis-matching requests and limits Update documentation.
Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) I can work on this, I am wondering if we should do the same for CRI(like docker) and CNI processes.
Sure, if you want to work on this go ahead. I think we should probably try to make this be toggleable via a kubelet flag. I believe there are other windows specific flags already implemented in the kubelet (like one to run as a windows service IIRC)
docker/containerd should probably have a flag that allows them to run as high priory as well but would likely be different issues in the respective projects
/reopen I don't think this was ever fully addressed
Reopened this issue. <detailsIn response to : /reopen I don't think this was ever fully addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
also related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: govc import.vmdk kube.vmdk ./kube/ ``` <del> Verify that the VMDK was correctly uploaded and expanded to 10GiB: </del> <ins> Verify that the VMDK was correctly uploaded and expanded to ~3GiB: </ins> ```sh govc datastore.ls ./kube/

When wgetting to this address during the installation process, this just fails silently. Turns out the issue is that if you take a raw and curl that address you get a TLS error. I've fixed it by changing to , but obviously that's more risky.
Related (filed after an email exchange around wget) I don't have easy access to the vSphere so I can't verify how it relates to my debian install.
Yes, that is exactly the error I was experiencing.
Does curl work? Or is it busted too?
Curl is not installed on the image by default. I'm building a new image based on Debian 7.7 where I pull in curl by default, so we can switch to using that everywhere (instead of an awkward mix of wget and curl).
Thanks for reporting this I hadn't seen this yet. I uploaded a new VMDK that includes curl and updated the deployment scripts to use curl. I verified that deployment works again for me.
I can confirm that this works now with latest build from master on ESX. Probably still won't work with v0.4.3 (even w new image) since the scripts still used wget back then.
We are probably going to fork for v0.5 soon. But I can backport if this trips folks up. sent from tiny keyboard On Nov 12, 2014 9:09 AM, "Ben Smith" wrote:


Question: function config-ip-firewall { echo "Configuring IP firewall rules" <ins> # Do not consider loopback addresses as martian source or destination while # routing. This enables the use of 127/8 for local routing purposes. sysctl -w net.ipv4.conf.all.route_localnet=1 </ins> # The GCI image has host firewall which drop most inbound/forwarded packets. # We need to add rules to accept all TCP/UDP/ICMP packets. if iptables -L INPUT | grep "Chain INPUT (policy DROP)" > /dev/null; then

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Fluentd pod failed to become ready on master after upgrade from 1.8-HEAD. Logs from https://k8s- Multiple CIs are hitting this: https://k8s- https://k8s- https://k8s- Anything else we need to know?: From the timeline, possibly caused by the fluentd hostnetwork change:
https://k8s- has the same symptom as well. /sig instrumentation
Could you please take a look?
/unassign /assign
It doesn't seem like an upgrade issue actually. It happens exclusively on master and I see it happening just after regular cluster startup. In the prom-to-sd logs: Why is metadata server unaccessible from the master? Can it be related to the metadata-proxy?
Yeah I agree, edited the title. Sorry for the confusion.
This is definitely metadata proxy related. Are you still seeing this? I introduced a bug in that I fixed in
https://k8s- Same issue today. Again on master
Okay. My guess is that the metadata proxy isn't getting scheduled on the master for some reason. Looking ...
Hm. The master's shows that it's got the right node label and that a proxy is being scheduled:
Okay, I'm seeing the same thing on a newly-created HEAD cluster. Metadata-proxy is running and serving requests ... my next guess is that the iptables rules are messed up.
The iptables rules are fine:
cc
Note that we moved fluentd out of host network recently. Why do we want to do that?
Thanks, yup. We're trying to root cause; unclear if that's what caused it, or a change in the metadata proxy.
Thanks for debugging this! found that on master node is 0, which should be 1. I set it to 1, and it does fix the issue.
I've included the fix in
Thanks a lot for debugging! Could you please add to the PR description? Also, what about previous branches? Is it going to work on 1.7 and 1.8 branches, or we need to propagate your fix there?


Question: "UsageBytes":      bounded(10*e2evolume.Kb, 80*e2evolume.Mb), "WorkingSetBytes": bounded(10*e2evolume.Kb, 80*e2evolume.Mb), "RSSBytes":        bounded(1*e2evolume.Kb, 80*e2evolume.Mb), <del> "PageFaults":      bounded(100, 1000000), "MajorPageFaults": bounded(0, 10), </del> <ins> "PageFaults":      bounded(100, expectedPageFaultsUpperBound), "MajorPageFaults": bounded(0, expectedMajorPageFaultsUpperBound), </ins> }), "Accelerators": gomega.BeEmpty(), "Rootfs": ptrMatchAllFields(gstruct.Fields{

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --- ci-crio-cgroupv1-node-e2e-conformance many Many tests failing, they seem to be timing out. Example: Flakiness began late on 07/21 with this failure run: Note that cri-o jobs do not pin version of cri-o or Fedora CoreOS so it is possible the underlying components got bumped. /sig node
/cc
/milestone v1.22
I will take this /assign
/triage accepted
/remove-kind flake /kind failing-test as stats/summary test is now continuously failing
it is hard to tell because the tests were very flaky before, but I am suspicious of for causing the perma-failures. I have opened to test, where we can compare the difference in failures. It's possible this is just due to some crio configuration not being happy with the metrics server bump
/milestone v1.23
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale /remove-kind failing-test /kind flake back to flaking:
it looks like the major page fault value is a bit off. I've opened to fix


Question: "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/sets" <ins> utilflag "k8s.io/apiserver/pkg/util/flag" </ins> clientset "k8s.io/client-go/kubernetes" kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm" kubeadmapiext "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha1"

/kind feature What happened: When calling , there is no way to override default flag values for the different components of the controlplane. This is surprising, as in the config structure (and in controlplane phase codebase itself), there are explicit settings for doing that (APIServerExtraArgs, ControllerManagerExtraArgs, SchedulerExtraArgs). What you expected to happen: Those config fields should be exposed as flags on and How to reproduce it (as minimally and precisely as possible): Try to deploy K8s using Kubeadm with a custom Admission Control setting (I'll follow up on this issue with a PR)
btw, I am not sure which sig owns kubeadm.
it's sig-cluster-lifecycle.
/sig sig-cluster-lifecycle
/sig cluster-lifecycle


Question: } } } <ins> // this mimics how resource/get.go calls the customcolumn printer func TestIndividualPrintObjOnExistingTabWriter(t *testing.T) { columns := []printers.Column{ { Header:    "NAME", FieldSpec: "{.metadata.name}", }, { Header:    "LONG COLUMN NAME", // name is longer than all values of label1 FieldSpec: "{.metadata.labels.label1}", }, { Header:    "LABEL 2", FieldSpec: "{.metadata.labels.label2}", }, } objects := []*v1.Pod{ {ObjectMeta: metav1.ObjectMeta{Name: "foo", Labels: map[string]string{"label1": "foo", "label2": "foo"}}}, {ObjectMeta: metav1.ObjectMeta{Name: "bar", Labels: map[string]string{"label1": "bar", "label2": "bar"}}}, } expectedOutput := `NAME      LONG COLUMN NAME   LABEL 2 foo       foo                foo bar       bar                bar ` buffer := &bytes.Buffer{} tabWriter := printers.GetNewTabWriter(buffer) printer := &printers.CustomColumnsPrinter{ Columns: columns, Decoder: legacyscheme.Codecs.UniversalDecoder(), } for _, obj := range objects { if err := printer.PrintObj(obj, tabWriter); err != nil { t.Errorf("unexpected error: %v", err) } } tabWriter.Flush() if buffer.String() != expectedOutput { t.Errorf("nexpected:n'%s'nsawn'%s'n", expectedOutput, buffer.String()) } } </ins>

Compare the following two outputs (note the bad alignment in the last line): The custom-columns string is exactly the same in both cases, but something messes up the column widths when using openapi print columns. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Columns printed when using are misaligned. What you expected to happen: Columns in all rows should be aligned. How to reproduce it (as minimally and precisely as possible): Deploy Service Catalog, patched with Run the above two commands Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-15T23:06:17Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-10-17T15:09:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} (same behavior with kubectl built from current master branch)
/sig api-machinery
/remove-sig api-machinery /sig cli PR is ready:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: cmd.Flags().String("resource-version", "", "If non-empty, the labels update will only succeed if this is the current resource-version for the object. Only valid when specifying a single resource.") usage := "Filename, directory, or URL to a file identifying the resource to update the labels" kubectl.AddJsonFilenameFlag(cmd, &options.Filenames, usage) <ins> cmd.Flags().Bool("dry-run", false, "If true, only print the object that would be sent, without sending it.") </ins> return cmd }

kubectl should support a --dry-run option for every mutation to show what it would do, without actually doing it We might need API changes in order to make this high-fidelity. For instance, if we wanted to know what a pod config would look like after admission control populated defaults, we'd need to execute most of the creation path.
For generator types commands, I think there is a usability argument that the default when a user specifies should be to dry-run (and a user should be allowed to override that with ). I think I'd rather encourage composition of commands (pipe) than every generator command in the system creating objects. We may also want to clarify dry-run against the server vs dry run against the client - there are lots of places where we want to generate locally to a file or an editor or a pipe command before we want to create it. At this point, OpenShift has a few commands that treat -o  as "dry-run" - new-app (complex generator) and export (never expect to send it to the server). We also have a few that are designed to be compositional in a pipe always - process (take a template and turn it into a list of api objects), which never creates the objects for you (you have to pipe to ). I'm somewhat partial to having and support the presence of changing the default of to true, because that's more common and easier than having to set and . The counter argument is that is the preferred action, and is format only - except that by definition --dry-run using the default output format isn't very useful (you created a service named foo and an rc named bar). My gut is that most users once they understand what a particular generator does their default path is always going to be to review the change, and then if they want to create it they can pipe to or just by appending to their existing command. I'm not sure that's a strong enough argument to couple --dry-run and -o, but based on our experience so far it's going to be the "normal" path.
:+1: for this feature. In particular, I would like an ability to validate an object definition separate from actually creating the object. e.g. running at the validation step of my deployment pipeline.
I agree that the validation use case is important. I would distinguish --dry-run from --local.

The former would still entail contacting the server and returning the result of the mutation. The latter would imply that only local information were used.
- I'm planning to add a parameter like 'pretty' or 'labelselector' to API requests. Once the server get this parameter, it will do validation to the resource and check if request can pass admission controls, but return success before write to etcd. I've read the documented about API changing, it lists some principles when changing the api object, but parameters are not mentioned. Is there anything need to be taken special care of when adding a parameter?
We need to be careful about dry-run with admission control... some admission controllers (like quota, specifically), record things to etcd in the process of admitting an object. Also, dry-run wouldn't work with a list of objects intended to be created in order, where later objects reference earlier objects (and those references are validated at create time)
We could potentially carry "dry run" down into the admission controller request path. However, a query parameter may not be appropriate, instead there's probably a HTTP header that already covers this. We try not to mix query parameters and post bodies in many cases. , Jordan Liggitt wrote:
- Thanks for advices! I'm trying to make myself thoroughly familar with request handling process. And during this period, I may need more of your help. I think I've replied this message before but unfortunately it's not showed up here:-( - Can you provide a link to the code about the HTTP header? Thanks!
This feels like a variant of an header with a sub parameter - possibly or possibly content-type . , Victor HU wrote:
+1 for this. My use case is to understand the difference between a stack file and an existing cluster, but this command would be a substitute.
It looks like support for has been . I think this issue can be closed.
We still need to make touch the server, or at least decide that we don't want to.
you mean move server side?
has just released kubeval: From to sig-apps mailing list: I am not sure if it validates the values and other constraints checked on the server-side. Just leaving it here as it might be relevant to the feature discussion. (h/t briangrant for pointing this out).
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
a HTTP header that already covers this. We try not to mix query parameters and post bodies in many cases. Do you remember why you said that? I'm trying to understand ...
should fix this issue, we can close it when the server-side dry-run feature reaches beta. The feature above () will fix this use-case, and even validate against the admission chain. is being built on top of server-side dry-run, and will be a good way to visualize the output of dry-run apply:
This is now beta.


Question: if !ok { return fmt.Errorf("Service %s not in cache even though the watcher thought it was. Ignoring the deletion.", key), doNotRetry } <ins> return s.processLoadBalancerDelete(cachedService, key) } func (s *ServiceController) processLoadBalancerDelete(cachedService *cachedService, key string) (error, time.Duration) { </ins> service := cachedService.state // delete load balancer info only if the service type is LoadBalancer if !wantsLoadBalancer(service) {

BUG REPORT: Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: CC: What happened: There are resource leaks in the cloud with respect to PublicIp and LoadBalancer Frontend Ip Pool and HealthProbe What you expected to happen: All created cloud resources, associated with a deleted service, should be destroyed. This is causing the Azure Load Balancer to fill up its frontend IP Pool which is currently limited to 10 entries. After a few resource leak, the new kuberntes services cannot get valid External IP since the Azure Load Balancer is full. How to reproduce it (as minimally and precisely as possible): Create 5 namespaces and deploy the guestbook-all-in- example. (making sure you turn on the LoadBalancer type for the frontend service) Wait for all of the 5 PublicIP to be provisioned in the Azure Load Balancer. Delete all of the 5 namespaces using the Poll the namespaces until they are totally gone from kubernetes and recreate them. Redeploy the guestbook-all-in- in all 5 namespaces. You will probably end up with 5 new IPs BUT not all of the previous ones will be deleted. Anything else we need to know: Kube-Controller-Manager Logs Service IP after the first pod creation Service IP after the re-creation Azure frontend Pool Screen Shot <img width="799" alt="screen shot 2017-03-27 at 22 17 38" src=""
The following PublicIP were never removed
Does this repro on a single master cluster?
Yes, to ensure that all the logs were in the same file, I stopped all services on the other two kube-master. (e.g. kube-controller-manager, kube-apiserver, kube-scheduler) The log might still show entries when the configuration was setup as multi master
Update: I have retested with the deletion of the service/deployment instead of the namespace. I still have the same issue. As you can see 3 IP should have been delete from Azure.

52.167.217.196 52.167.223.67 52.167.222.61 <img width="721" alt="screen shot 2017-03-28 at 18 34 55" src=""Find the Debug Log for the kube-controller-manager here (start from the last Service Restart Entry)
I'm seeing that the controller is calling delete for only , which seems to indicate that the controller only processed a single delete callback for the load balancers. I don't have an azure cluster handy. -- if it's possible, can you repro with: Start with empty cluster Ceate guestbook example Delete guestbook example, but don't create again. see if the IP resources leak, and post the kube-controller-manager logs. I would like to understand if this occurs without having to recreate the objects immediately.
"delete the guestbook example" == "delete the guestbook namespace" or "manually delete the guestbook's svc"? (I only ask to clarify because I suspect that if simple "single service delete" were failing, we'd see a lot more noise/issues about it)
I can confirm that if I wait after the deletion that all of the cleanup job occurs there is no leak. The problem is when the deleted service is recreated with the same name under the same namespace.
The creation needs to occur while the deletion of the resource in Azure is still ongoing. The apiserver returns that the resources are gone and cleaned which allow us to recreate.
Can you post the kube-controller-manager log for the simplified repro? (create -delete -create) with one copy of the guestbook app)?
let me extract that information from the logs. I created test4 last, deleted it, then recreated in a continuous sequence. without deploying anything else at the same time.
(Not to interrupt any diagnosing, but FYI: I tried to intentionally use the value from the cloudprovider-levlel in order to avoid any sort of conflict like this: That value is used to name the LB rules/probes and NSG rules, with the hope that it would avoid these sorts of conflicts. But I'm not sure how is formed... if it's deterministic based on the names of things, then I can imagine there could be some conflicts occurring.)
do you have a preferred logging level ? it seems that level 10 logs too much... The journal output is missing a bunch of lines.

My original Log looks better and doesn't have missing lines. I think the first one was set at level 3
Here are the Azure Logs for the first test where test5/frontend's IP leak for example azureloadbalancer The azureloadbalancer with test5/frontend only shows thats the cloud provider was involved only for the creation and not the deletion. If we look at the we can see an example of the logs when the azure_loadbalancer gets called for deletion Adding a bit of logs from the controller related to The First time the Service was created and configured in Azure was at this time and line number () The second time it was created was at this time and line number () All of the specific logs below
the UID you use looks different between each delete/create First create/delete for test1/frontend Second Create
I think that the answer lies between line and in the first log file I attached from the OP
More investigation on my side For the 's service, we see this in the log when the controller manager tries to see if it needs to delete the service For the 's service we see the following instead The main difference between the two are those two lines  When it cannot delete the code is here. Which simply seems to return false, and ignores the deletion of the service all together. The weird part that I'm not too sure why, is that the deletion failures occur before the namespace was recreated and repopulated with resources. this is the last trace I can find for the deletion of the test5 namespace at the Namespace and its content get recreated here, at which is almost 1m30m later
I guess the difference can be explained by the that looks global for the client. The first pass, it get sets to which then trigger the exit right away.... So from the code, it just seems that the call to ` returns nothing....
and I have reproduced the issue with the smallest possible interaction. The following manifest (2 services with type:LoadBalancer) was used The Logs of the kube-controller-manager at lvl4 logging level from the time range of the test only. The IPs after the second create are The following IP and FrontEnd configuration leaked The commands executed were PS: Note that I was not able to reproduce with a manifest containing a single service object. As soon as I the second to the manifest, I was able to reproduce the issue right away.

Azure's LB snapshot <img width="788" alt="screen shot 2017-04-06 at 17 38 43" src=""
So this happens even outside of the namespace deletion? Is there anything obvious in the k-c-m logs? I'm very curious to know if this is a bug in the ServiceController or in the Azure cloudprovider code.
Also, do you see any errors in the Audit Log in the Portal for this resource group?
there are no issues in Azure' audit log nor the Activity logs for this RG. Regarding your question about the source of the issue, I would tend to say that the issue is in the ServiceController. Here's my reasoning. For the service we have the following lines Creation 1 Deletion Creation 2 Now for the service we have the following lines Creation 1 Deletion Creation 2 As you can see there is not even a call from the class to issue a delete based on this log entry ,hence the class is never called. Now, as to why the class forgets about the deletion and moves on to the recreation. So far, my best bet is that the deletion process and handled one at the time, and since it takes a while for the IP to be removed in Azure, the logic in the detect that frontend2 was created and is in need of an IP. Right before the frontend2 second creation I see this. Also, the order of creation/deletion is creates test/frontend creates test/frontend2 deletes test/frontend creates test/frontend2 creates test/frontend
So I looked at the code and I see that my hypothesis was true. (if I'm not mistaken) Here's the break down of my analysis. The Cloud-controller-manager calls the command with ConcurrentServiceSyncs hardcoded to (it cannot be change either from the command line) The service controller creates a single worker thread that uses a workingQueue to get the list of services to process. The worker's sole purpose is to synch the Service. Upon the deletion of both services, the queue gets filled up with two entries: and The worker pops out and starts processing the deletion. It takes around 3mins. Before the end of the deletion of , both services are recreated and the queue gets repopulated. After is deleted, is removed from the queue to be processed.

In the method, the service is queried to see if it is present Since it was readded (with the same name) there are no Error returned, hence no deletion requested. From here, I'm not 100% sure what would be best. what is your take here ? should the SynchService stop querying for the Service existance in the apiserver and be given the Instruction to add/remove the service directly from the Informer's DeleteFunc method? Knowing that there is only a single worker. The queue could look something like this So all of the creation and deletion would be process in the right order. The only problem I see with this right now is that there seem to be a retry mechanism that could re-queue a delete action after a recreate... I guess the safest bet is really have a way to prevent the apiserver from accepting the same service to be created if the CloudProviderPublicIP has not been removed... This is going back to this Issue I guess(). Even though the issue is requesting that the user be able to query the deletion Status, here we would want the apiserver to not delete the service from his list until it is fully deleted...
This looks to be me like an A-B-A () problem with the cloud resource. Let me do some digging...
This looks to be another instance of If the reconciliation was performed periodically, the IPs will be garbage collected.
could you explain me what the reconciliation would provide to the serviceController ? I've read the issues you linked, but I'm still not 100% sure I understand the proposed logic. (probably because I still don't know how the service controller fully works -- mostly with respect to caching)
The idea is there would be a continual loop over the cloud load balancers to see if there are any configured that should be there. If there is a mismatch, the load balancer will be deleted. Admittedly, this is not a hugely satisfying answer. It's along the lines of: there is no sync up of the Kubernetes state with the cloud provider state in the service controller and this can lead to stranded/inconsistent resources.
I see. thanks for the explanation. What do you think about having the master prevent the creation of a service that was previously deleted until the cloud controller manager is done with his clean up job ? (e.g.

all configuration are deleted in the cloud's LB) That should cover the use case where services are deleted and recreated almost back to back.
I still don't even understand why there's a conflict when they're deleted/recreated back to back. The Service UUID should be different. Or is it because the SerivceController stops trying to delete the previous service because ServiceController only tracks by name and not by the UUID?
I guess either way there's still agreement that this is a bug in ServiceController layer... please tell me if this isn't the case and we need to be more actively engaged from the Azure side.
-- as far as I can tell, it is tracked by name and not UUID
correct, the key used to track consist of the string:
This seems to be a result of using the namespace/service name in . I can reproduce this easily by causing the ServiceController to get busy syncing a new service and then deleting and recreating a different service like this: In the controller-manager I see logs like this (I'm using the Openstack provider in this case): You can see that the ServiceController essentially "lost" the delete because the name was the same. Because the calling syncService does not work concurrently (here ), by the time it finished handling and started on , the service had already been recreated. So when sync service checked it (here ) it already existed again and dropped into the case instead of the case like we would want. The end result is that the not only the public IP but the cloud provider's load balancer is orphaned.
Any updates here?
No time to take this up in this release -- punting.
/sig azure
/area platform/azure
remove from 1.7 milestone
Move to next milestone per sig/network's request.
any chance this could be picked up for 1.8 ?
I'll take a stab at this... Note I'm going to remove the sig/azure since this seems to be a global issue in the
I've run into this today (I think - at least the symptoms seem to be the same, with services perpetually stuck in for public IP, and I have been creating, deleting and recreating services perhaps too quickly). Is there a workaround?

(I saw this problem on a test cluster, so I've resorted to deleting the cluster and re-creating it for now, but I'd like to avoid that in the future if I can, since it takes a while... Also, it's not done yet, so I don't know if it actually solves my problem - but if it doesn't, my problem is likely not related to this issue.) When I do there's only three IP:s listed for my subscription, so I know I'm not hitting the limit of simultaneously provisioned IP:s (the defaults allow 20) - do you know of other limits I could be hitting? Are there other resources except the public IP addresses that leak, that could be subject to similar caps? I'm running k8s 1.9.2 on the client, and 1.8.7 on the cluster.


Question: } fs.IPVar(&s.BindAddress, "insecure-bind-address", s.BindAddress, ""+ <del> "The IP address on which to serve the --insecure-port (set to 0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces).") </del> <ins> "The IP address on which to serve the --insecure-port (set to 0.0.0.0 or :: for listening in all interfaces and IP families).") </ins> // Though this flag is deprecated, we discovered security concerns over how to do health checks without it e.g. #43784 fs.MarkDeprecated("insecure-bind-address", "This flag will be removed in a future version.") fs.Lookup("insecure-bind-address").Hidden = false

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are flag descriptions where is stated that the unspecified address behaves differently depending on the IP family: However, that's not the behavior implemented in and , clarified in the golang documentation What you expected to happen: Fix the comment to state the reality, i.e How to reproduce it (as minimally and precisely as possible): From the root of the kubernetes repo Anything else we need to know?: xref Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind cleanup /help wanted /good-first-issue /priority important-longterm /cc
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/assign /remove-help
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
For the flag in kube-proxy, is used if the address is not specified and can't be resolved by hostname, so I suggest to remove this comment: 0.0.0.0::` for all IPv6 interfaces'
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I am very much in favor of this. Being correct and precise in these flags is valuable. To convince myself you were right, I wrote this program: And then I ran it for various inputs: In these flags and docs, we should be clear that if you are specifying a v6 addr and port you have to use syntax. Or maybe we should just audit and fix ALL such places to support and document that "nothing" is how you specify all interfaces. E.g. --bind-address=":80" is more obvious than either "0.0.0.0:80" or "[::]:80".

Even if those syntaxes work, let's just make "" the norm?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details