Question: shouldError:      false, }, { <del> name:                  "cloud provider is external", nodeIP:                netutils.ParseIPSloppy("10.0.0.1"), nodeAddresses:         []v1.NodeAddress{}, externalCloudProvider: true, </del> <ins> name:              "cloud provider is external", nodeIP:            netutils.ParseIPSloppy("10.0.0.1"), nodeAddresses:     []v1.NodeAddress{}, cloudProviderType: cloudProviderExternal, </ins> expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.0.0.1"}, {Type: v1.NodeHostName, Address: testKubeletHostname},

I am reproducing this in OpenShift using the OpenStack cloud provider, but nothing here is specific to either OpenShift or OpenStack. When migrating a cluster which uses in-tree cloud provider to use external cloud-provider, we use the following process: cloud controllers in KCM CCM kubelets with --cloud-provider=external Between steps 1&2 the node-lifecycle, routes, and service controllers are not running anywhere, but this situation isn't unsafe for the cluster and we can keep it short. Between steps 2&3 is more interesting. The node controller is running in CCM, and the kubelets are running with --cloud-provider=openstack (or whatever, just not external). The main loop of node-controller will ignore a newly created node because . However, the loop does not have a similar exclusion. . Note that we shortcut if the kubelet is using an external cloud provider, but not if we don't. So between steps 2&3 both kubelet and CCM's node controller are managing Node.Addresses. If the legacy cloud provider and external CCM don't return the same list of NodeAddresses this causes flapping. Running OpenShift on OpenStack we have 2 problems. Firstly, our legacy cloud provider and external CCM produce different lists. We don't think this would be a problem in practise, except for the second problem. Kubelet adds the annotation to the node containing the node IP. This allows the CCM node controller to apply the same filtering based on node IP as kubelet does for the legacy cloud provider. This means that, although the lists are different, the node IP is still listed first which is the only thing that matters in practise. The problem is that kubelet only adds the annotation when --cloud-provider=external which, between steps 2&3, it is not. This means that Node.Addresses flaps between a correct and a broken configuration, as the NodeAddresses returned by the external CCM cannot be filtered by node IP without the annotation. The ideal solution here would be to ensure that only 1 controller is managing Node.Addresses during this period in the same way we do for Node creation with the external CCM taint. This would fix the issue entirely.

A fix with a much smaller scope would be to have kubelet add the node IP annotation unconditionally so, although Node.Address can potentially flap, it will at least be between 2 valid configurations. Given that this should only ever be a problem during the upgrade window, the smaller fix may be more expedient. Node.Addresses should remain valid at all times when kubelet is configured with --cloud-provider=openstack but the CCM is also running. Good luck No response 24 OpenStack (but as described, not OpenStack-specific) No response No response No response No response
/sig cloud-provider
we've an agenda item for next week's (11 May) SIG cloud provider meeting about this topic
/triage accepted
Would it work to swap steps 2 and 3?
Good question. I don't think so. That would leave us without the Node, Node lifecycle, and Service controllers until all kubelets were updated. This would mean: No cluster scale-up or scale-down No creation, deletion, or modification to LB Services I think we could handle this in OpenShift, although it's a bit fragile and I suspect updating all the kubelets can take a really long time on a big cluster. However, I think this would break cluster-api or any deployment tooling which rolls out updates by creating new machines. I think this would never complete: the new nodes would never come up because the node controller isn't running.
I am still trying to understand the issues here while reviewing your pr: , but I agreed with your comments above: That would leave us without the Node, Node lifecycle, and Service controllers until all kubelets were updated. This would mean:


Question: to update the golden list of tests stored in `testdata/`.  Changes to that file require review by sig-architecture. <del> To update the list, run ```console bazel build //test/conformance:list_conformance_tests cp bazel-bin/test/conformance/conformance.yaml test/conformance/testdata ``` </del> <ins> To update the list, run `hack/update-conformance-yaml.sh` </ins> Add the changed file to your PR, then send for review.

The currently provided command for generating requires bazel, the command should be updated to one which does not require bazel Currently: Suggested: /sig testing /sig architecture /area conformance
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle rotten /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
We're going to need to update this to verify under as well instead of under the bazel "unit tests", ref: I'll work on this.
done in
This is wonderful to see. Thank you


Question: if err := wrapped.SetUpAt(dir, fsGroup); err != nil { return err } <ins> if err := volumeutil.MakeNestedMountpoints(s.volName, dir, *s.pod); err != nil { return err } </ins> data, err := s.collectData() if err != nil {

This bug affects any pod that has a container with a nested volume (any type) beneath an API data volume, such as a secret, configMap, downwardAPI or projected volume. In the description and reproduction details below I use a secret volume, but any of the four will demonstrate this behavior. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a pod with nested secrets volumes (e.g. one volume mounted at and one mounted at ), the container fails to start with an error like: What you expected to happen: The container should start with both and mounted. How to reproduce it (as minimally and precisely as possible): out master, any version since a file with 2 secrets and a pod that makes volumes for the 2 secrets, with one of them mounted within the other: the 2 secrets and the pod. Assuming you named the above file : Anything else we need to know?: This may work for other runtimes if they create mountpoints outside of the sandbox. But it's definitely broken with Docker as the runtime.
This should be fixed by /sig storage /sig node /assign


Question: "sigs.k8s.io/structured-merge-diff/v4/fieldpath" "k8s.io/apiextensions-apiserver/pkg/apiserver/schema/cel" <ins> structurallisttype "k8s.io/apiextensions-apiserver/pkg/apiserver/schema/listtype" </ins> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/validation/field"

doesn't do the same embedded resource and x-kubernetes-list-type "map" and "set" invariant validation that happens in custom resource create/update validation. Does that mean embedded objects and list-type fields under custom resource status can contain invalid data? if so, that needs to be fixed in a way that tolerates existing invalid data (ratcheting validation) Originally posted by in /sig api-machinery /kind bug /priority important-longterm
/triage accepted
I would like to collaborate for fixing the validation issue for custom resources and tolerate the existing invalid data. Could you please guide me with some code pointers or files and elaborate on how can I proceed for fixing this bug.
I haven't tested this, but something along the lines of this: we'd need tests to exercise the following scenarios on a CRD that specifies a status subresource with a list property with x-kubernetes-list-type=map and x-kubernetes-list-type=set a custom resource instance that already has duplicate items in the status property list can be updated successfully while still having duplicate items in the list (validation doesn't block an invalid → invalid status update) a custom resource instance that already has duplicate items in the status property list can be updated successfully to set unique items in the list (validation doesn't block an invalid → valid status update) a custom resource instance that does not have duplicate items in the status property list cannot be updated to set duplicate items in the list (validation blocks a valid → invalid status update)
since it's been a couple of months, i'm going to try and take a stab at this /assign
/assign
Hi k8s 1.25 Bug Triage Shadow here, just checking in to see if this is still on track for k8s 1.25? Thank you.
has stalled needing tests and a rebase
Thanks for the update.
Hello with the code freeze approaching, is this still on track for v1.25? Thanks!
not really, the status on is the same (needs tests)
/milestone clear


Question: o.ApplyLeaderElectionTo(o.ComponentConfig) c.ComponentConfig = *o.ComponentConfig } else { <del> cfg, err := loadConfigFromFile(logger, o.ConfigFile) </del> <ins> cfg, err := LoadConfigFromFile(logger, o.ConfigFile) </ins> if err != nil { return err }

I want to be exported as a public function imports scheduler code and runs scheduler simulations for pods to decide which nodegroup should be scaled up. consider this case: a user is running a scheduler with custom scheduler config the user wants cluster-autoscaler to run scheduler simulations based on the custom scheduler config For ii default scheduler config is used in cluster-autoscaler scheduler simulations. Right now, it is not possible to use custom scheduler config for cluster-autoscaler scheduler simulations but I am working on a PR () to support this feature. For this feature, I need to parse scheduler config. . It would help if we can have exported as (just like ) so that programs which need to parse scheduler config can use the exported function directly instead of writing custom code.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
PR:
/assign vadasambar
wdyt?
/sig autoscaling
LGTM, thanks.


Question: HealthzBindAddress: kubeletHealthzBindAddress, HealthzPort:        utilpointer.Int32Ptr(constants.KubeletHealthzPort), RotateCertificates: kubeletRotateCertificates, <ins> ResolverConfig:     resolverConfig, </ins> }, }, },

What happened: TestKubeletDefault unit test fails when run locally on system that has systemd-resolved active What you expected to happen: Unit test succeeds How to reproduce it (as minimally and precisely as possible): Use ubuntu with systemd-resolved active. Run Output: Anything else we need to know?: Environment: Kubernetes version (use ): Master branch (v1.19.0-alpha) Cloud provider or hardware configuration: n/a OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /priority important-soon I created this issue to track what happened. I'll have a PR to fix soon. /assign
/area kubeadm /sig cluster-lifecycle
/remove-sig node
thanks for reporting. /remove-kind bug /kind failing-test /assign


Question: "time" ) <ins> type timedStoreDataSlice []timedStoreData func (t timedStoreDataSlice) Less(i, j int) bool { return t[i].timestamp.Before(t[j].timestamp) } func (t timedStoreDataSlice) Len() int { return len(t) } func (t timedStoreDataSlice) Swap(i, j int) { t[i], t[j] = t[j], t[i] } </ins> // A time-based buffer for ContainerStats. // Holds information for a specific time period and/or a max number of items. type TimedStore struct { <del> buffer   []timedStoreData </del> <ins> buffer   timedStoreDataSlice </ins> age      time.Duration maxItems int }

I have some reports of a Node reporting a a resource capacity for cpu as a negative value. Any idea why cpu could report as a negative value? Is this an expected normal outcome? In its current state, it prevents any pods from being scheduled to that Node. Tips on how to debug further are appreciated. - any ideas?
Can you check machineInfo on Cadvisor or kubelet on the node?
a.k.a:
It is a bug in cAdvisor side. I am assigning it to you :-)
here is contents of /proc/cpuinfo Pertinent pieces of a log:
Machine is AWS instance if that matters....
We build the topology from system/cpu info. I think we can return partial topology information and ignore caches if its not present. This isn't a recent change though. Is it happening on just a single machine? I'll file a cAdvisor issue.
Can you link the referenced issue once created?
Re-assigning to since he's working on google/cadvisor
Could be this was caused by getting a Node on EC2 with old hypervisor, and I am the first victim...
That sounds likely. Most of the cache information is there in /proc/cpuinfo. So looks like the sysfs bits are disabled.
Can you try this cadvisor image on your machine and check /api/v1.3/machine output: sudo docker run  --volume=/:/rootfs:ro  --volume=/var/run:/var/run:rw  --volume=/sys:/sys:ro  --volume=/var/lib/docker/:/var/lib/docker:ro  --publish=8080:8080  --detach=true  --name=cadvisor  jnagal/cadvisor:canary
hmm
ah, selinux :( You'll need to run it with --privileged, I think. who's been dealing with it.
No luck with --privileged either on this box...
This has to do with SELinux, I think we had to disable it for this to work (some of our mounts are being denied). We're working on a better solution... This is not a problem in Kubernetes since we don't run Kubelet/cAdvisor in a Docker container.
ok, unfortunately, i have managed to kill docker on the box in question, and i cannot get the daemon to restart right now so I am not sure when that box will be available to debug against again.

I can provide an environment with a similar issue if required.
The fix is checked in to head. Can you build the binary and try?


Question: 12.34.56.78	another.domain # Entries added by HostAliases. <del> 123.45.67.89	foo 123.45.67.89	bar 123.45.67.89	baz 456.78.90.123	park 456.78.90.123	doo 456.78.90.123	boo </del> <ins> 123.45.67.89	foo	bar	baz 456.78.90.123	park	doo	boo </ins> `, }, }

What would you like to be : Currently, specifying multiple aliases for a single host using results in a line for each alias in . Example: leads to an file containing something like Why is this needed: Having multiple entries for the same IP address in can break DNS resolution for some software. See for more details <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/sig node


Question: # but we only need to rebuild targets if the contents actually changed.  That # is what the .stamp file represents. $(foreach dir, $(ALL_GO_DIRS), $(META_DIR)/$(dir)/$(GOFILES_META)): <del> FILES=$$(ls $</*.go | grep -v $(GENERATED_FILE_PREFIX));   </del> <ins> FILES=$$(ls $</*.go | grep --color=never -v $(GENERATED_FILE_PREFIX));   </ins> mkdir -p $(@D);                                            echo "gofiles__$< := $$(echo $${FILES})" >$@.tmp;          cmp -s $@.tmp $@ || touch $@.stamp;

Using the grep from OSX, fails when GREPOPTIONS/GREPCOLOR is set as shown below (). Workaround : unset GREPOPTIONS/GREPCOLOR Note that using GNU grep using brew install will show that GREPOPTIONS shows immediately that GREP_OPTIONS may be interfering with make ()
oh, here's badly manged conversion-
Will look, thanks. For make-related bugs, please @ me directly.
Will do. thanks
This breaks building kubernetes inside Alpine Linux:
I'm pressing more and more on the idea that the only supported build is the dockerized build. Trying to accomodate every random environment is just too taxing. , Marc Villacorta wrote:
I agree with We can support the stuff in to run on mac to get you dockerized but everything else is only supported in there. I've worked around a few BSD/OSX things in but it isn't fun. We should try to keep that stuff isolated.
I agree, with the usual caveat that we need to be doing all of the build in one container with as little as possible happening outside. OS X tends to have old and broken versions of important programs, which is terrible. We discussed setting up CI for it but it's not going to happen in the near term. Alpine/Busybox purposefully omit features from their versions for good reasons, but it's still more work to support than it's worth.
Ok, I will re-think my workflow.
How did you find yourself building in Alpine, in I may ask? , Marc Villacorta wrote:
nothing is forcing me to use Alpine I just like the distro. I run stuff on CoreOS and my aim is to solve the build-deliver-run pipeline with a single small container. By the way, I have the problem installing the GNU package :-)


Question: . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> storagev1 "k8s.io/api/storage/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" clientset "k8s.io/client-go/kubernetes"

created a spread sheet of the slowest running tests for the pull-kubernetes-gce job. cc discussion on slack: spreadsheet: pie chart of test durations from SIGs: ! it seems appropriate to evaluate the state of the tests and reduce these runtimes. possibly move tests to periodic by marking them as [Slow]. /assign /kind design /area test (?)
/remove-kind bug
Are pull jobs the right place to be calculating the time? Or should be be counting in CI instead?
pull is the one timing out, so probably?
Looking at some of the top slowness:: subPath should fail for new directories when readOnly specified in the volumeSource: This test case is no longer defined in master HostPathType: This test case has not been checked in yet and is trying to be in a PR PersistentVolumes-local StatefulSet with pod affinity should use volumes spread across nodes when pod has anti-affinity: This test is already marked as slow in master
Created This should address most of the top slow storage tests


Question: // Code generated by go-bindata. // sources: <ins> // translations/OWNERS </ins> // translations/extract.py // translations/kubectl/OWNERS // translations/kubectl/de_DE/LC_MESSAGES/k8s.mo

/assign /cc Please ensure that every sub directory of / contains an owners file. The following sub directories do not define them: The result of this is that any PR which changes anything in api, Godeps, staging, vendor, etc will always get assigned to one of you for both the review and approval.
I recall someone also mentioning that certain files or dirs may not need approvers files. If that would help, we can add that functionality (i've already started the work)
staging and api should be easy to find owners for. may have ideas for translations. The big one is Godeps, third_party, vendor. Those are just grunt work. And figuring out criteria for approving is hard as they are a lot of code and it is unclear that things are tested well or automatically. (I'm thinking things like libraries for accessing cloud APIs). Having a dedicated team/process there might help.
I'll open some proposals for staging and API.
Godep in particular needs to be watched for things like licenses, which I have mostly been doing.
Thanks Clayton for the staging and api additions! How about if we pick two of you at random and shove them in as owners for each of the remaining directories?
What's left: translations: the kubectl specific translations is being handled by sig-cli, but the rest of the files like README and test have been handled and approved mostly by thirdparty: this is a tricky mix of stuff like forked code.. mostly handled by the likes of or for approvals. Godep/vendor: These could probably be solved by having a process to follow, and then delegating to a team in OWNERSALIASES for both folders.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Still relevant. /remove-lifecycle stale


Question: v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/watch" "k8s.io/client-go/tools/cache" watchtools "k8s.io/client-go/tools/watch" </ins> "k8s.io/kubernetes/test/e2e/framework" <ins> e2enode "k8s.io/kubernetes/test/e2e/framework/node" </ins> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" testutils "k8s.io/kubernetes/test/utils"

New TestGrid alerts at since commit/. Found 2 failing rows in node-kubelet-serial-containerd: 2 new, 0 other 2 new Test Failures in node-kubelet-serial-containerd. Test has failed 3 times since it last passed at commit/. It started failing between commit/ (Fri Mar 24 12:02 PDT) and commit/ (Fri Mar 24 16:03 PDT): . Test has failed 3 times since it last passed at commit/. It started failing between commit/ (Fri Mar 24 12:02 PDT) and commit/ (Fri Mar 24 16:03 PDT): . Not sure suspect No response /sig node release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake /remove-kind failing-test See
/remove-sig release
Taking a look at The test failed due to: so it means that the pod was not terminated and deleted properly in the test that ran prior. The is the one creating this pod, but I would expect that as part of the test teardown, the namespace is deleted (and thus the pod should be deleted).
The services log show that the kubelet never sent the request to delete the pod, that is strange...
From the build-logs, the test execution for when the pod was created: Kubelet logs for the pod - Full logs are here -
Here's the termination part of the logs:
It looks like to me the issue here is that the kubelet is being restarted in a bit of an unfortunate time resulting in at least two related issues: The pod status has been updated on the api-server to terminal just prior to the kubelet restarting. The status manager did not yet complete . Then after kubelet restarts and comes back up, it attempts to add this pod , (even though it is in terminal phase). This seems like the first issue: if the pod phase is terminal on the API server, we probably should not add it back.

The pod worker receives the , but the pod worker quickly terminates (). I am assuming this is because of , i.e. based on pod cache lookup the pod is terminal. The side affect of this is that the status manager never receives the signal that the pod is (which is normally done as part of . This is because in this case, we never call which sets to be true. As a result, I think will never return true, so this pod will never be able to be deleted.
I have made a test which will reproduce this behavior in a node e2e, see It can be reproduced by starting a pod and then deleting it and triggering kubelet restart at the right time. A watch is setup to trigger a kubelet restart as soon as the pod enters the succeeded phase. If the kubelet restart happens before the kubelet is able to delete the pod on the API server, but after the phase is updated, then we trigger the issue. The pod will be re- to kubelet after restart, but will never be deleted on the API server despite it having a deletion timestamp set and being in terminal phase. Here are logs from the and running the above repro node e2e test. Here is the kubelet logs portion prior to the restart: and after restart: It also looks like we are hitting another invalid state in based on the log assertion: I tried the same test on 1.26 and it passes. This is because on 1.26, the status manager phase to terminal happens at roughly at the same time when we delete the pod. On 1.27+, after , the status phase to terminal can happen earlier and the status manager can skip deleting the pod because . In this case, after start, will never be true, so the pod never will get deleted.
But in general pods can be in terminal phase and not be deleted (like plain pods and pods from jobs). They would eventually be deleted by the pod GC otherwise. Is this a bug in the device plugin?
That is true, but the issue here is that the pods have a deletion timestamp set as well. If the deletion timestamp is set and the kubelet is up, it is the kubelet's responsibility to terminate and delete those pods.

This doesn't have anything with device plugin, rather we caught the issue due to the device plugin test which asserts that all pods are terminated and deleted prior to starting the test.
I have a WIP fix for this issue in
That makes sense
Should we aim 1.27 with this fix? This seems like an important regression.
I discussed this with yesterday and we agreed that this can wait until 1.27.1 Clients should be able to handle terminated pods. Worst case scenario, the pod GC would take care of these pods eventually.
Cherrypicked to v1.27 (to be released in 1.27.2) in


Question: ], embed = [":go_default_library"], deps = [ <ins> "//pkg/apis/scheduling:go_default_library", </ins> "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/github.com/stretchr/testify/assert:go_default_library", <del> "//vendor/github.com/stretchr/testify/require:go_default_library", </del> ], )

What would you like to be : Currently, in the following public methods do not have unit test coverage: - - These methods are great candidates for adding unit test coverage. Why is this needed: Adding unit test coverage will both ensure they are working correctly and (more importantly), encode their expected behavior in the test suite. Additionally, if we want to change their behavior in the future, it'll be helpful to already have the test suites. Extra note I think this could be a great candidate for someone who is new to contributing to sig-node/k8s. I'm happy to mentor any first time contributors :) Please respond saying you'd like to work on this issue and I'll assign it to you. Please reach out to me on k8s slack (where I'm also if I can be of any assistance.
/sig node
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, I would like to work in one of these.
/assign Awesome!
Hi, I am a first time contributor and would be happy to contribute to one of these.
Hi, I want to help if still need it.


Question: # "burstable" category so the kubelet doesn't backoff from restarting it. limits: cpu: 100m <del> memory: 200Mi </del> <ins> memory: 170Mi </ins> requests: cpu: 100m <del> memory: 100Mi </del> <ins> memory: 70Mi </ins> livenessProbe: httpGet: path: /healthz

Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?


Question: # "burstable" category so the kubelet doesn't backoff from restarting it. limits: cpu: 100m <del> memory: 200Mi </del> <ins> memory: 170Mi </ins> requests: cpu: 100m <del> memory: 100Mi </del> <ins> memory: 70Mi </ins> livenessProbe: httpGet: path: /healthz

Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?


Question: in.AlgorithmSource.DeepCopyInto(&out.AlgorithmSource) in.LeaderElection.DeepCopyInto(&out.LeaderElection) out.ClientConnection = in.ClientConnection <del> out.DebuggingConfiguration = in.DebuggingConfiguration </del> <ins> in.DebuggingConfiguration.DeepCopyInto(&out.DebuggingConfiguration) </ins> if in.BindTimeoutSeconds != nil { in, out := &in.BindTimeoutSeconds, &out.BindTimeoutSeconds *out = new(int64)

What would you like to be : Change the external DebuggingConfiguration member variables to pointer types. Why is this needed: So that we can set their default values to something different from golang's, for example enable profiling by default. /sig api-machinery /cc
The label(s) cannot be applied. These labels are supported: <detailsIn response to : What would you like to be : Change the external DebuggingConfiguration member variables to pointer types. Why is this needed: So that we can set their default values to something different from golang's, for example enable profiling by default. /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
done in


Question: "fmt" "strings" <ins> "github.com/davecgh/go-spew/spew" </ins> "github.com/golang/glog" )

We now have pretty structured way of validating api object, but the error message to end user is not descriptive, e.g. when creating a replication controller with restart policy 'never', we get: even as simple as label name, we shouldn't assume end-user is familiar with our requirement. Apart from what is wrong, we'd better also give why it is wrong.
The main thing wrong here is not having a useful String() method. and have both been pretty adamant about not attaching code to API objects, but here's a clear UX justification. What if we only attached methods to internal API objects, not versioned ones? Also: this should print %#v, rather than %+v, which would be a BIT more readable.
Orrrr we could switch to spew, which is already a dep.
Actually we're printing a reflect.Value there, which makes no sense at all. I will fix. EDIT: no we aren't, I can't read-- will still fix though


Question: import ( "context" "fmt" <del> "io/ioutil" "net/http" "strconv" "strings" </del> <ins> "net" </ins> "time" <ins> "github.com/onsi/ginkgo" </ins> v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" <del> "k8s.io/kubernetes/test/e2e/framework" "github.com/onsi/ginkgo" </del> <ins> "k8s.io/kubernetes/test/e2e/framework" </ins> imageutils "k8s.io/kubernetes/test/utils/image" ) const ( <del> testPodPort = 8080 testProxyPort = 31235 // Firewall rule allows external traffic on ports 30000-32767. I just picked a random one. </del> <ins> testPodPort    = "8080" noSNATTestName = "no-snat-test" </ins> ) var ( testPod = v1.Pod{ ObjectMeta: metav1.ObjectMeta{ <del> GenerateName: "no-snat-test", </del> <ins> GenerateName: noSNATTestName, </ins> Labels: map[string]string{ <del> "no-snat-test": "", </del> <ins> noSNATTestName: "", </ins> }, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { <del> Name:  "no-snat-test", </del> <ins> Name:  noSNATTestName, </ins> Image: imageutils.GetE2EImage(imageutils.Agnhost), <del> Args:  []string{"no-snat-test", "--port", strconv.Itoa(testPodPort)}, Env: []v1.EnvVar{ { Name:      "POD_IP", ValueFrom: &v1.EnvVarSource{FieldRef: &v1.ObjectFieldSelector{FieldPath: "status.podIP"}}, }, }, }, }, }, } testProxyPod = v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "no-snat-test-proxy", }, Spec: v1.PodSpec{ HostNetwork: true, Containers: []v1.Container{ { Name:  "no-snat-test-proxy", Image: imageutils.GetE2EImage(imageutils.Agnhost), Args:  []string{"no-snat-test-proxy", "--port", strconv.Itoa(testProxyPort)}, Ports: []v1.ContainerPort{ { ContainerPort: testProxyPort, HostPort:      testProxyPort, }, }, </del> <ins> Args:  []string{"netexec", "--http-port", testPodPort}, </ins> }, }, }, } ) <del> // Produces a pod spec that passes nip as NODE_IP env var using downward API func newTestPod(nodename string, nip string) *v1.Pod { pod := testPod nodeIP := v1.EnvVar{ Name:  "NODE_IP", Value: nip, } pod.Spec.Containers[0].Env = append(pod.Spec.Containers[0].Env, nodeIP) pod.Spec.NodeName = nodename return &pod } func newTestProxyPod(nodename string) *v1.Pod { pod := testProxyPod pod.Spec.NodeName = nodename return &pod } func getIP(iptype v1.NodeAddressType, node *v1.Node) (string, error) { for _, addr := range node.Status.Addresses { if addr.Type == iptype { return addr.Address, nil } } return "", fmt.Errorf("did not find %s on Node", iptype) } func getSchedulable(nodes []v1.Node) (*v1.Node, error) { for _, node := range nodes { if !node.Spec.Unschedulable { return &node, nil } } return nil, fmt.Errorf("all Nodes were unschedulable") } func checknosnatURL(proxy, pip string, ips []string) string { return fmt.Sprintf("http://%s/checknosnat?target=%s&ips=%s", proxy, pip, strings.Join(ips, ",")) } </del> // This test verifies that a Pod on each node in a cluster can talk to Pods on every other node without SNAT. // We use the [Feature:NoSNAT] tag so that most jobs will skip this test by default. var _ = SIGDescribe("NoSNAT [Feature:NoSNAT] [Slow]", func() {

What would you like to be : The test currently only works on Cloud Providers or environments that set the External IP on nodes, because it is using the framework directly to check the IPs of the pods, using a no-snat-test-proxy to check if the Pods are not using SNAT. However, this can be simplified, removing the no-snat-test-proxy Pod and executing the queries directly in a Pod. Why is this needed: The test should work in all environments, and it's failing https://k8s-,%20master&include-filter-by-regex=NoSNAT
/sig network /help wanted /failing test
/remove kind-feature /kind failing-test
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove kind-feature
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I would like to investigate this problem. /assign
I have filed , how could I verify if my patch fixes the problem?
I run them locally with KIND


Question: **Does this PR introduce a user-facing change?**: <!-- <del> If no, just write "NONE". </del> <ins> If no, just write "NONE" in the release-note block below. </ins> If yes, a release note is required: Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required". -->

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The PR template is a little unclear to new contributors. EG: If I understand what "uncomment" is, and leave it on it's own line I get: which has two problems: I probably left the space after the (so the bot regex won't match) I probably didn't put a newline before the uncommented line, which is necessary for it to not just continue to be a comment block: What type of PR is this? /kind cleanup vs What type of PR is this? /kind cleanup Another issue is in this bit: release-note release-note NONE If you're not an experienced contributor, it's not very clear where you should write "NONE" and what kind of quoting you should use, etc. What you expected to happen: I hope for the PR template to be a little more obvious and easy to use How to reproduce it (as minimally and precisely as possible): create a new PR Anything else we need to know?: /sig contributor-experience cc Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/remove-kind cleanup
/assign
GitHub didn't allow me to assign the following users: paralika. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to work on this issue.
was an improvement but many of the original observations still stand. Here's what I found looking at 75 latest PRs (~67 human-written ones if you discount 8 "Automated cherry pick"): The instruction to "uncomment only one /kind line" seems to suggest keeping all the other options present with prefix, but few people do that! I found just 7, of which: 2 PRs retained that whole section of the template, not picking a kind at all.

1 of these still got kind label magically set by the bot (possibly from linked issue?) I confirmed (tip: ⋯ -Quote reply lets you see the markdown source) these were NOT people that removed but forgot newline, getting bitten by markdown "lazy continuation" rule — they retained on all lines. 3 PRs uncommented 1 line. 2 PRs uncommented multiple lines, resulting in multile kind/... labels! Example: I don't know if you'll want to encourage "Uncomment one or more", or want to keep it a "hidden feature" for experienced users? However, from the (secondary) angle "template teaches people how the project/infrastructure works", the "only one" wording is slightly misleading — it obscures the fact /kind just sets a label, similar to /area, /sig etc. (and from what I saw, multiple areas or SIGs are common?) One hypothesis is people feel leaving the rest is noise that detracts from writing & reading the description? Weak evidence that's not the only motivation: a few (didn't count, I'd say ~3) retained the instructions line, which is clearly noise, e.g.: I don't think we can measure this, but see below. Found 3 of these, these are actual failures of the process! ~5% of people took the effort to choose a kind but didn't understand the "uncomment" instruction :cryingcatface: (bot assigned kind/cleanup anyway, not sure how)
It seems delete-all-irrelevant mechanic is better than "uncomment" mechanic. It avoids markdown pitfall, the end result has less noise, and most people choose it anyway. What if the template had them without prefixes? This would prevent mistakes of the last kind, where prevented bot from seeing contributor's choice! A downside is that when somebody is lazy / misses this step, they'll create PRs with ALL kind/ labels :worried: Is the bot smart about /commands inside fenced blocks? Or HTML comments? If yes, could use them as a signal that you made a conscious choice: I've also seen projects that use checklists in templates: downsides: Would require bot support to understand it.

It's slightly worse on the "template teaches people how the project/infrastructure works" angle — you won't learn from that these are regular /commands you can just write wherever. Misses on benefits of "delete" mechanic. People will likely delete anyway, with same mild confusion about two-ways-to-do-it. Well, this was too many words to suggest such small tweak :laughing:, but: exactly what to do, rather than vague "uncomment". Not sure if this will reduce mistakes of the last kind. space between avoids risk of leaving space mentioned in OP, without having to explain it. removing the rest =less worries, no markdown pitfall.
OK, that answers the question about bot & fenced blocks :grin: ^^^^^ /remove-kind api-change /remove-kind cleanup /remove-kind documentation /remove-kind failing-test /remove-kind feature
/remove-kind design /remove-kind flake


Question: if e, a := item.expectErrorPod, gotPod; !reflect.DeepEqual(e, a) { t.Errorf("%v: error pod: wanted %v, got %v", i, e, a) } <ins> if e, a := item.expectForgetPod, gotForgetPod; !reflect.DeepEqual(e, a) { t.Errorf("%v: forget pod: wanted %v, got %v", i, e, a) } </ins> if e, a := item.expectError, gotError; !reflect.DeepEqual(e, a) { t.Errorf("%v: error: wanted %v, got %v", i, e, a) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pod got stuck in Pending state forever when failed to bind to host. What you expected to happen: For it to be rescheduled on another host How to reproduce it (as minimally and precisely as possible): Not always reproducible. But simply created an RC replicas=2 on a 3 host setup, and one of the Pods got stuck in pending state for a while. Following error messages were found in the scheduler logs: Describe pod output: RC yml: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.1 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): RHEL7, Docker native 1.12.6 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scheduling
Can you provide the details steps for this issue? And the output of ?
The log just showed the node was notReady or unreachable. Seems the kubelet node is not quite stable that the connections with the master always get lost.
Another pod of the same RC (replicas=2) was started successfully on one of the nodes. I'd expect the failed one to at least be attempted to start again, may be on the node where its peer is running (as there are no anti affinity rules defined on it). But it was stuck in pending state forever (and there was no other reference for the failed pod besides the original log I've attached). I was able to create new RCs of the same kind on the same set of nodes successfully after the Pod scheduling failure, so it looks like some nodes were available for allocation. Perhaps there are some other logs I can look at /provide? no definite steps.

We just extensive validation tests for kubernetes were a bunch of RCs/Deployments/Services/Ingress controllers get created. And starting k8s 1.7.x started observing this failure. Once we see it again, I'll fetch the node status and update the bug.
From the log, it seemed that failed to bind your pod to the node and also failed to forget the pod, which led to unable to reschedule the pod to other nodes. Would you please append when you start and reproduce this issue. Also the kubelet log of that erroneous node.
^^ could you start the schedule with the above param next time for validation test run?
/cc
I ran our tests again, and hit this issue with several pods: There were a few more. Since none of them is assigned to a node. I included logs for all kubelet's. I also included scheduler log with : :
Seeing this as well. hypothesis: broke the error handling when fails. Before that commit, when failed it would log an error and pass the pod to the error handler (), which is in charge of retrying scheduling the pod. After that commit, when fails it skips the error handling and scheduling is never retried. I'm doing some experiments with patching that and I think there's more to this bug than just that (for example why is failing in the first place? is the scheduler cache corrupted? why?), but that's what I've got so far.
/cc
Can you post the description of per namespace?() Are you enabled ? And what's the config of component?
there are no specific parameters passed on the scheduler start besides address and cloudconfig
I posted a patch that i believe fixes this issue at -- for us it's resolved the issue so far, but I'm not very reliably able to reproduce so it's a bit hard to check. could you apply and see if your validation tests pass?
thanks!! We will be able to validate it either today or early next week
Are you able to validate the fix provided by through ? Just saw this issue, and I think it is a serious regression in 1.7 release and we should patch it. cc/ the patch manager for 1.7 patch release.
our QA have been busy with the current release, and didn't have a chance to validate the fix yet. Hopefully this week.

Patching sounds like a great idea. It is a regression indeed as we haven't hit this issue in the previous versions of k8s with the same set of validation tests.
Thanks everyone who helped investigate this!
just ran validation test against the patched branch - no pods stuck in Pending any more, so the fix did the job! thank you
I have this problem with k8s v1.7.0 . How can I fix it, upgrading to v1.7.X? Thank you!
I am seeing this in 1.9. What steps I need to take to solve this problem?
I am seeing this periodically on Azure VMs
any chance you could open a new issue linked to this one with the requisite troubleshooting info (versions, etc.) and label it with ` Thanks!


Question: // newDefaultFieldManager is a helper function which wraps a Manager with certain default logic. func newDefaultFieldManager(f Manager, typeConverter TypeConverter, objectConverter runtime.ObjectConvertor, objectCreater runtime.ObjectCreater, kind schema.GroupVersionKind, ignoreManagedFieldsFromRequestObject bool) *FieldManager { <del> f = NewManagedFieldsUpdater(f) f = NewStripMetaManager(f) f = NewBuildManagerInfoManager(f, kind.GroupVersion()) f = NewCapManagersManager(f, DefaultMaxUpdateManagers) f = NewProbabilisticSkipNonAppliedManager(f, objectCreater, kind, DefaultTrackOnCreateProbability) f = NewLastAppliedManager(f, typeConverter, objectConverter, kind.GroupVersion()) f = NewLastAppliedUpdater(f) return NewFieldManager(f, ignoreManagedFieldsFromRequestObject) </del> <ins> return NewFieldManager( NewLastAppliedUpdater( NewLastAppliedManager( NewProbabilisticSkipNonAppliedManager( NewCapManagersManager( NewBuildManagerInfoManager( NewManagedFieldsUpdater( NewStripMetaManager(f), ), kind.GroupVersion(), ), DefaultMaxUpdateManagers, ), objectCreater, kind, DefaultTrackOnCreateProbability, ), typeConverter, objectConverter, kind.GroupVersion()), ), ignoreManagedFieldsFromRequestObject, ) </ins> } // DecodeManagedFields converts ManagedFields from the wire format (api format)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --pull-kubernetes-integration runkubectldiff_tests It only shows failure cases in recent 2 days. The last date is I tried to reproduce it locally, but can't
/sig testing
/sig cli
/cc
I’ve only briefly taken a look at this, but my understanding is that server-side diff is flaking because the timestamp is different (by like a second) between the one already applied and the diff apply. do you think this could be related to this recent change Could delaying when we update the timestamp cause it to sometimes get misaligned with what’s being diffed? I’m not super familiar with diff, but it feels like it should ignore differences between timestamps that get updated on each call, right? Is there some sort of mechanism we have in that knows to ignore timestamps that just isn’t being done for fields?
This error probably shows that we have a path that causes the object to be re-written to etcd when it shouldn't.
As we can see, the diff only shows a difference in the timestamp but not in the fields that are managed. We are not supposed to update the timestamp if the managed fields haven't changed. That timestamp only gets updated if the two steps happen in different seconds (that's why it's flaky). That is definitely not a flake but an actual failure.
Failed 81/236 times today in /priority critical-urgent I think, correct it if I am wrong.
/triage accepted
correct. who should be assigned as the owner of this?
I can take it, I'm looking into it now
/assign


Question: <dc:format>image/svg+xml</dc:format> <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> <del> <dc:title></dc:title> </del> <ins> <dc:title /> </ins> </cc:Work> </rdf:RDF> </metadata>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --AWS EBS CSI driver returns topology segments It should merge the topology segments Enabling , and then describe the I have found in the docs which indicates that the duplication is by intention Also I found this and pull
/assign
/sig storage /tirage accepted /assign
It's caused by the migration from CSI to in-tree PV. AWS EBS CSI driver provides PV with: CSI translation changes -and we end up with two same items in the array. It makes sense to squash these duplicates.
cc what do you think?
Why is aws csi driver provides PV with Seems to me this is not needed.
I believe aws csi driver recently changed to using kubernetes topology labels instead of vendor-specific topology labels. Note that the elements of a matchExpression are together. I'm not sure if that's the correct approach if the intention is to eventually get rid of the vendor-specific topology label. cc
I think the topology label is defined by the csi driver? Is it okay to get rid of the vendor specific label?
The intention was to put both labels on all nodes to maintain backwards compatibility with existing PVs. but provisioned PVs should only have one or the other, putting both labels on the PV is a bug: Now that we've released the bug and some PVs have both labels, it will be hard to remove the vendor specific label.
thanks for your follow up. After the provisioned PV is assigned to a node, removing the duplicate label would still be an issue ?
It depends on which label is going to be removed. I'd suggest to remove the in-tree one (), because we can be sure that nodes have the other label from registering the CSI driver.
+1
/triage accepted /assign
fixed by Closing


Question: <dc:format>image/svg+xml</dc:format> <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> <del> <dc:title></dc:title> </del> <ins> <dc:title /> </ins> </cc:Work> </rdf:RDF> </metadata>

The logo (, ) font does not match the others. I believe the png/pdf versions are correct.
What other ones? It should be the same font - they look the same to me...


Question: // use a channel to ensure we don't look at the metric before it's // been set. ch := make(chan struct{}, 1) <ins> longestCh := make(chan struct{}, 1) </ins> mp.unfinished.notifyCh = ch <ins> mp.longest.notifyCh = longestCh </ins> c.Step(time.Millisecond) <-ch mp.unfinished.notifyCh = nil if e, a := .001, mp.unfinished.gaugeValue(); e != a { t.Errorf("expected %v, got %v", e, a) } <ins> <-longestCh mp.longest.notifyCh = nil </ins> if e, a := .001, mp.longest.gaugeValue(); e != a { t.Errorf("expected %v, got %v", e, a) }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: To reproduce using
/sig api-machinery


Question: } type DriverCapabilities struct { <del> Attach         bool `json:"attach"` SELinuxRelabel bool `json:"selinuxRelabel"` </del> <ins> Attach          bool `json:"attach"` SELinuxRelabel  bool `json:"selinuxRelabel"` SupportsMetrics bool `json:"supportsMetrics"` </ins> } func defaultCapabilities() *DriverCapabilities { return &DriverCapabilities{ <del> Attach:         true, SELinuxRelabel: true, </del> <ins> Attach:          true, SELinuxRelabel:  true, SupportsMetrics: false, </ins> } }

kind feature What happened: Currently there is no mechanism to report metrics as the flex plugins are not enabled to collect metrics. All flex volumes lack this feature as compared to in-tree/built-in drivers even though it is possible to collect such metrics for most of the plugins. What you expected to happen: Depending on the availability of the metrics from the mounted FS, flex plugins should allow to report metrics for the FS. How to reproduce it (as minimally and precisely as possible): The default behavior that can be proved easily by just running a command similar to the following command on a node that is running a flex volume. This will show that there are no metrics collected for flex volumes. kubectl proxy $curl 127.0.0.1:8001/api/v1/nodes/<IP/proxy/stats/summary Environment**: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: local x86 OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux KubeStorage 4.4.0-130-generic -Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Others: /sig storage
+1 Yes, it will be good, if flex volume can support atleast the default metrics( i.e ), which most of the volume plugins support now. Based upon flex plugin "capabilities" parameters, it can be decided, whether the plugin supports "default metrics" or "nil metrics".


Question: var addressType v1.NodeAddressType if strings.HasPrefix(i.HardwareAddr.String(), MAC_OUI_VC) || strings.HasPrefix(i.HardwareAddr.String(), MAC_OUI_ESX) { <del> addressType = v1.NodeExternalIP } else { addressType = v1.NodeInternalIP </del> <ins> v1.AddToNodeAddresses(&addrs, v1.NodeAddress{ Type:    v1.NodeExternalIP, Address: ipnet.IP.String(), }, v1.NodeAddress{ Type:    v1.NodeInternalIP, Address: ipnet.IP.String(), }, ) </ins> } <del> v1.AddToNodeAddresses(&addrs, v1.NodeAddress{ Type:    addressType, Address: ipnet.IP.String(), }, ) </del> glog.V(4).Infof("Find local IP address %v and set type to %v", ipnet.IP.String(), addressType) } }

updated the automatic reloading logic from file watch to polling. This change in behavior needs to be documented [ ] Update the v1.29 release notes to call out this change in behavior (ref: ) to replace references to file watch with polling. /sig auth /triage accepted /assign nilekhc


Question: if mirrorPod.DeletionTimestamp != nil || !kl.podManager.IsMirrorPodOf(mirrorPod, pod) { // The mirror pod is semantically different from the static pod. Remove // it. The mirror pod will get recreated later. <del> klog.Warningf("Deleting mirror pod %q because it is outdated", format.Pod(mirrorPod)) if err := kl.podManager.DeleteMirrorPod(podFullName); err != nil { </del> <ins> klog.Infof("Trying to delete pod %s %v", podFullName, mirrorPod.ObjectMeta.UID) var err error deleted, err = kl.podManager.DeleteMirrorPod(podFullName, &mirrorPod.ObjectMeta.UID) if deleted { klog.Warningf("Deleted mirror pod %q because it is outdated", format.Pod(mirrorPod)) } else if err != nil { </ins> klog.Errorf("Failed deleting mirror pod %q: %v", format.Pod(mirrorPod), err) <del> } else { deleted = true </del> } } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: of a static pod (really the mirror pod) results in multiple attempts to re-create the mirror pod before it finally succeeds (note that in some runs it goes into Running or Terminating state -- it just depends upon the exact timing of the . Note that in the below trace the pod name is elided from the output. Note that the actual static pod continues to run as normal. What you expected to happen: Mirror pod should start without additional state transitions How to reproduce it (as minimally and precisely as possible): Create a static pod, it, and Anything else we need to know?: Environment: Kubernetes version (use ): 1.14/OpenShift 4.2 (master has the same code) Cloud provider or hardware configuration: AWS (not relevant) OS (e.g: ): Not relevant Kernel (e.g. ): Not relevant Install tools: Not relevant Network plugin and version (if this is a network-related bug): N/A Others:
/sig node
More details: The bug appears to be a race condition between deleting the old mirror pod, the API server deleting its record of it, and creating a replacement that's not related to skew. I suspect the reason I didn't see it bouncing back and forth in my earlier tests was simply that I wasn't running the kubectl get node frequently enough; if I run it continuously, I see it bounce. The sequence is: 1) Somebody kubectl deletes the mirror pod. 2) The static pod code in (around line 1630) notices that the pod has been marked deleted (its deletion timestamp is not nil and it is a mirror pod), deletes it (in DeleteMirrorPod in , via the apiserverClient), and creates a new one (again, via the apiserverClient). 3) The next time around, the static pod code may see either the old pod (that has been deleted) or the new one as the mirror pod.

If the old pod is the one that's seen, it sees that there's a deletion timestamp, so it deletes it again -- which, since pod deletion is by name (UID isn't involved) results in the new pod being deleted and yet another new pod being created. If the new pod is the one that's seen, then we break out with the spin cycle complete. Still TBD is why the mirror pod isn't getting replaced right away. But here are traces of the relevant sequences: Bad sequence: Good sequence: The messages at lines 7 and 18 were printed immediately after the call to apiServerClient...Create(). But the key is at lines 9 and 19. At line 9, note that the UID (...) is the UID of the old mirror pod, which has a deletion timestamp and therefore the mirror code deletes the pod -- but the pod that's actually in existence at the time is the new pod. At line 19, however, note that the mirror pod is now the new one, which is not marked for deletion, and is therefore left alone. The solution (I'll open a PR shortly) is to check both the pod name and the UID to make sure that we're in fact trying to delete the pod we think we want to delete.
Thanks for the detailed issue :) Really awesome!
I'm seeing this issue on an upgrade from v1.14.4 to v1.15.1 following the on a CentOS 7 system. After installing the updated kubelet package and restarting kubelet per the instructions, the cluster went into a loop of continually restarting the etcd pod from one control plane node and the scheduler pod from another control plane node. This continued for approximately 10 minutes before the etcd pod finally stayed up. It was another 5 minutes before the scheduler pod finally ran. I noticed v1.15.2 was released although the change log doesn't show this issue included. Was there a plan to back-port this fix to 1.15 and 1.14 (not to the same extent, but I have seen this behavior on a 1.14.4 cluster)? Since this behavior can happen to the apiserver pod, it has the potential to be very disruptive to a cluster.
The actual etcd pod, or the mirror pod (which is more likely)?

I hadn't specifically planned on backporting it, but it shouldn't be a problem; this code has been stable.
Ah, good point--the actual pod does appear to remain running. That makes it less important than I was initially thinking. However, it is a bit disconcerting for someone who happens to be looking at the state of the cluster (or for monitoring) to see the pod appear to be stuck in a restart loop. If it were just a few iterations, it might not matter too much. However, I'm seeing this happen for minutes at a time. For example, to verify that the real pod remained available I did a on the apiserver pod for one master, and output showed it in the restart loop for 15 minutes before it settled down. IMO that's worth backporting if it's low risk.
I opened a cherrypick against 1.15 for this.
Appreciated. Thanks!


Question: driverName: driverName, } if shouldTimeout { <del> timeout := plugin.PluginClientTimeout + 30*time.Millisecond </del> <ins> timeout := plugin.PluginClientTimeout + time.Second </ins> fakeDRADriverGRPCServer.timeout = &timeout }

test Since Oct 25th No response stack trace: Test has been flaking with master golang as well. /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
seems like we need to increase the timeout.. /assign
I tried with different values of timeout in function viz and running test binary using stress tool. In most cases, test is getting passed.
Flake in master blocking board: /cc
Flake in as well.
is waiting for approve been a while, which try to fix the flake
We can still observe this flakiness with 30ms timeout in CI Wondering, if timeout has to be increased further. Please let us know if more information is required.
It flaked in today as well. /reopen
Reopened this issue. <detailsIn response to : It flaked in today as well. </details
We're seeing this flake frequently downstream as well, both in TestUnprepareResources and TestPrepareResources.
cc
is trying to fix it but still pending


Question: // manageJob is the core method responsible for managing the number of running // pods according to what is specified in the job.Spec. // Does NOT modify <activePods>. <del> func (jm *JobController) manageJob(activePods []*v1.Pod, succeeded int32, job *batch.Job) int32 { </del> <ins> func (jm *JobController) manageJob(activePods []*v1.Pod, succeeded int32, job *batch.Job) (int32, error) { </ins> var activeLock sync.Mutex active := int32(len(activePods)) parallelism := *job.Spec.Parallelism jobKey, err := controller.KeyFunc(job) if err != nil { utilruntime.HandleError(fmt.Errorf("Couldn't get key for job %#v: %v", job, err)) <del> return 0 </del> <ins> return 0, nil </ins> } <ins> var errCh chan error </ins> if active > parallelism { diff := active - parallelism <ins> errCh = make(chan error, diff) </ins> jm.expectations.ExpectDeletions(jobKey, int(diff)) glog.V(4).Infof("Too many pods running job %q, need %d, deleting %d", jobKey, parallelism, diff) // Sort the pods in the order such that not-ready < ready, unscheduled

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? Job, resourcequota, scheduler Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-14T13:36:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: CentOS bare metal () Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Linux <server name3.10.0-514.2.2.el7.x8664 SMP Tue Dec 6 23:06:41 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: - Others: - What happened: Having created a resourcequota (e.g. 400M) and creating enough memory-limited jobs to go over that quota (e.g. 4 x 200M limit jobs), at first, two jobs are created and executed as expected, and two jobs fail at first due to memory quota: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 4s 4s 1 job-controller Warning FailedCreate Error creating: pods "memhog-y152c55u-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi After the first two jobs have been completed, instead of starting the remaining jobs as soon as possible, the remaining two jobs stay unstarted in this state for many, many hours until they finally start and complete.

First job: Start Time: Fri, 28 Apr 2017 15:00:08 +0300 One of the remaining 2 jobs: Start Time: Sat, 29 Apr 2017 07:13:34 +0300 What you expected to happen: I'm expecting the two waiting jobs to start executing as soon as the resources they need are available, e.g. as soon as the first jobs are completed. How to reproduce it (as minimally and precisely as possible): Bring up a kubernetes cluster, then execute these commands: $ kubectl create quota memoryquota --hard=memory=400Mi (cat /dev/urandom head -n 1)' --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory='2'm run- $ chmod +x run- $ for i in {1.4}; do ./run- 200 100; done job "memhog-0ju2eiuy" created job "memhog-t8qyt5sj" created job "memhog-ljaqz4ek" created job "memhog-eb8q8kr3" created Wait for around 30 seconds and the first jobs should be complete: $ kubectl get jobs NAME DESIRED SUCCESSFUL AGE memhog-0ju2eiuy 1 1 43s memhog-eb8q8kr3 1 0 42s memhog-ljaqz4ek 1 0 43s memhog-t8qyt5sj 1 1 43s $ kubectl describe jobs . However, the remaining jobs don't start at all as shown by: $ kubectl describe job memhog-ljaqz4ek Name: memhog-ljaqz4ek Namespace: test Selector: controller-uid=-2f00-11e7-8766- Labels: run=memhog-ljaqz4ek Annotations: run memhog-ljaqz4ek memhog -r100 100m --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory=200Mi,cpu=100m --record=true --image=derekway.

Parallelism: 1 Completions: 1 Pods Statuses: 0 Running / 0 Succeeded / 0 Failed Pod Template: Labels: controller-uid=-2f00-11e7-8766- job-name=memhog-ljaqz4ek run=memhog-ljaqz4ek Containers: memhog-ljaqz4ek: Image: derekwaynecarr/memhog Port: Command: memhog -r100 100m Limits: cpu: 100m memory: 200Mi Environment: <noneMounts: <noneVolumes: <noneEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 job-controller Warning FailedCreate Error creating: pods "memhog-ljaqz4ek-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi Furthermore, pods related to the remaining jobs aren't even created and only the completed pods are shown: $ kubectl get pods -a NAME READY STATUS RESTARTS AGE memhog-0ju2eiuy-bz0zm 0/1 Completed 0 7m memhog-t8qyt5sj-q9msw 0/1 Completed 0 7m Anything else we need to know: I've also created a stackoverflow question about this (), and asked about this in the kubernetes slack, where I was urged to create a bug report.
I will try to reproduce. The creation of a pod will be blocked by quota. In the absence of a pod, the scheduler has nothing to schedule, so it is not a scheduling bug. It's possible the issue is related to back-off in the job controller not attempting to create pods again. The event shows that it had been 3m since the job controller attempted to create pods again for the requisite job which implies some type of back-off. None of the controllers are quota aware, so I would anticipate they should just keep trying at reasonable intervals at this point. If you describe quota after the first 2 pods are completed, do you see quota usage being replenished as expected?
I would like to fix the bug, could you assigin it to me?

if pod create failed, jobcontroller.managejob() need return error, will requeue podcreate task.
-- maybe you have more insight, but it looks to me just reading the code that when a job controller is unable to create a pod due to lack of quota, its not actually an error that forces a re-queue of the job. in addition, i cant actually see when a full resync of the job work queue happens outside of the informer sync interval which is probably really long. see: see:
- feel free to open a pr, please cc me on your solution. i would like to understand from the job folks if there should be a periodic resync interval aside from the need to requeue the job when all pods cannot be spawned (which i assume your pr will handle via exponential backoff)
-- we will want to have a fix for this downstream as well.
Yes, after the first two jobs are complete, shows that 0M out of 400M is in use. I'm also able to create new jobs, up to the memory limit so 2 at a time, that execute as expected. Any jobs over that at a time will stay in the same waiting state.
- Yes, I want handle it via exponential backoff. IMO, MaxOfRateLimiter had implemented a periodic resync, we don't need another one.
I'll make sure to cherry-pick this to openshift ( fyi).


Question: } t.Logf("Found InstanceID(%s) = %sn", nodeName, instanceID) <del> instanceID, err = i.InstanceID(context.TODO(), nonExistingVM) </del> <ins> _, err = i.InstanceID(context.TODO(), nonExistingVM) </ins> if err == cloudprovider.InstanceNotFound { t.Logf("VM %s was not found as expectedn", nonExistingVM) } else if err == nil {

Hi, I am currently working on k8s cloud provider and I ran in pkg/cloudprovider package. Turned out that some providers code have unused variables and logic problems. Here is the report: session.New is deprecated some bugs and unused values oauth2.NoContext is deprecated, overwritten ctx and miss error checks missing error check unused values missing error check, loop logic issues Hope this report can help to cloud provider working groups. :)
/kind bug /sig cloud-provider /sig vmware
PR open for vsphere cloud provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func (csiNodeStrategy) Validate(ctx context.Context, obj runtime.Object) field.ErrorList { csiNode := obj.(*storage.CSINode) <del> // in 1.22, on create, set AllowLongNodeID=false // in 1.23, on create, set AllowLongNodeID=true </del> validateOptions := validation.CSINodeValidationOptions{ <del> AllowLongNodeID: false, </del> <ins> AllowLongNodeID: true, </ins> } errs := validation.ValidateCSINode(csiNode, validateOptions)

will be used as a preparation for relaxing CSI Node id length limitation, in 1.22, we need to make sure the AllowLongNodeID is set correctly according to the comment /sig storage /assign /cc
/milestone 1.22
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone 1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Can you make a better title - I almost jumped out of my skin :)
Oops, sorry about that lol
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: // absGidMin <= defGidMin <= defGidMax <= absGidMax absoluteGidMin          = 2000 absoluteGidMax          = math.MaxInt32 <del> heketiAnn               = "heketi-dynamic-provisioner" glusterTypeAnn          = "gluster.org/type" glusterDescAnn          = "Gluster: Dynamically provisioned PV" </del> linuxGlusterMountBinary = "mount.glusterfs" autoUnmountBinaryVer    = "3.11" )

several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.


Question: local -r ds_file="${dst_dir}/calico-policy-controller/calico-node-daemonset.yaml" sed -i -e "s@__CALICO_CNI_DIR__@/home/kubernetes/bin@g" "${ds_file}" fi <ins> if [[ "${NETWORK_POLICY_PROVIDER:-}" == "kube-network-policies" ]]; then setup-addon-manifests "addons" "kube-network-policies" fi </ins> if [[ "${ENABLE_DEFAULT_STORAGE_CLASS:-}" == "true" ]]; then setup-addon-manifests "addons" "storage-class/gce" fi

,%20google-gce all I can't say , but at least a few weeks ,%20google-gce No response No response /sig network
/assign
/triage accepted


Question: defer ctrl.Finish() cloud := GetTestCloud(ctrl) expectedVM := buildDefaultTestVirtualMachine("/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Compute/availabilitySets/AGENTPOOL1-AVAILABILITYSET-00000000", []string{}) <ins> expectedVM.Name = to.StringPtr("name") </ins> mockVMClient := cloud.VirtualMachinesClient.(*mockvmclient.MockInterface) mockVMClient.EXPECT().Get(gomock.Any(), "rg", "k8s-agentpool1-00000000-0", gomock.Any()).Return(expectedVM, nil) <ins> expectedNIC := buildDefaultTestInterface(true, []string{}) expectedNIC.VirtualMachine = &network.SubResource{ ID: to.StringPtr("/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Compute/virtualMachines/k8s-agentpool1-00000000-0"), } mockNICClient := cloud.InterfacesClient.(*mockinterfaceclient.MockInterface) mockNICClient.EXPECT().Get(gomock.Any(), "rg", "k8s-agentpool1-00000000-nic-0", gomock.Any()).Return(expectedNIC, nil) </ins> ipConfigurationID := `/subscriptions/sub/resourceGroups/rg/providers/Microsoft.Network/networkInterfaces/k8s-agentpool1-00000000-nic-0/ipConfigurations/ipconfig1` nodeName, asName, err := cloud.VMSet.GetNodeNameByIPConfigurationID(ipConfigurationID) assert.NoError(t, err) <del> assert.Equal(t, nodeName, "k8s-agentpool1-00000000-0") assert.Equal(t, asName, "agentpool1-availabilityset-00000000") </del> <ins> assert.Equal(t, "k8s-agentpool1-00000000-0", nodeName) assert.Equal(t, "agentpool1-availabilityset-00000000", asName) </ins> }

What happened: introduced new dependency on Availability Sets which is too restrictive for some existing installations: What you expected to happen: Azure cloud provider to successfully manage LBs, like it had been doing before. Instead, it died [with][1]: How to reproduce it (as minimally and precisely as possible): Fails every time in OpenShift CI. Can you point us to documentation around the infrastructure options that the Azure cloud provider expects for LB management? https://kubernetes- doesn't say anything about NIC names, and it doesn't have specifics around what things should look like without Availability Sets, or whether "without Availability Sets" is not supported (when previous iterations of the cloud provider had no problem with our lack of Availability Sets). Documenting cloud-provider assumptions about host hardware would have also helped avoid ; CC /sig cloud-provider /area provider/azure [1]:
/triage accepted
/assign
Notes on OpenShift's Azure infrastructure in ARM templates [here][1], in case that helps clarify the infra we use today. Those are OpenShift 4.6 docs, and 4.6 is based on Kubernetes 1.19, and things worked fine. The trouble we've been having when we took the same Azure-infra approach and tried to use it with Kubernetes 1.20, after openshift/kubernetes. [1]:
could you share an example of resourceID (please replace your subscription/resourceName with fake ones)?
Example CI run [here][1] from the NIC-renaming openshift/installer. Installer logs [here][2] include: I'm hoping that the subscription name is not sensitive, because we've been dumping them in public logs for a long time now . Should be IDs for any other resources you're interested in in those logs as well. [1]: [2]: https://gcsweb-
Looks like this is about the master node. Are you using standard LB or a basic one? For standard LB, the master node would be excluded from the LB by default.
[Standard][1] (and [here][2]). Is this documented somewhere? We've been using standard LBs [for years][3], with no problems until the bump to the Kube 1.20 Azure cloud-provider. [1]: [2]: [3]:
Are you referring to the built-in logic of Kube to never target master nodes with a service type LoadBalancer?

That used to be a problem (), but my understanding is that it has since been fixed.
Can we re-open this? kubernetes-sigs/cloud-provider-azure was [not sufficient][1] to get the cloud-provider happy on our infra, which again, contains no Availability Sets. There are some more notes about our infrastructure [here][2] (with discussion in [rhbz][3]), in case that helps. [1]: [2]: [3]:
who has a better grasp on this than me, also just opened . Sounds like fixing that broader issue might obsolete this ticket, because "the cloud-provider expects Availability Sets which OpenShift infra doesn't have in order to figure out which nodes to remove" doesn't matter if the cloud-provider isn't removing any OpenShift nodes in the first place.
can you please confirm the standalone VM scenario is supported? It's a critical use case for CAPZ (our control plane VMs in zone-enabled regions are not placed in availability sets)

What happened: introduced new regex which is too restrictive for some existing installations: This new restriction causes LBs to fail to sync if just one ipConf fails regex. What you expected to happen: This change was to enable optional behavior, any ipConf that fails regex should just be ignored. This will allow existing installs to work as they did prior to the change. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Suggest putting a continue here: This will restore previous functionality and appears to have no undesirable impact.
/sig cloud-provider /area provider/azure
Unclear if we should also put a continue here as well:
/assign
could you share an example of resourceID (please replace your subscription/resourceName with fake ones)? /triage accepted
Here's the error we're seeing: Just the string


Question: id="svg2" version="1.1" inkscape:version="0.48.3.1 r9886" <del> sodipodi:docname="kubernetes.svg" </del> <ins> sodipodi:docname="kubernetes_logo.svg" </ins> inkscape:export-filename="/home/thockin/Downloads/kubernetes.png" inkscape:export-xdpi="90" inkscape:export-ydpi="90">

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --AWS EBS CSI driver returns topology segments It should merge the topology segments Enabling , and then describe the I have found in the docs which indicates that the duplication is by intention Also I found this and pull
/assign
/sig storage /tirage accepted /assign
It's caused by the migration from CSI to in-tree PV. AWS EBS CSI driver provides PV with: CSI translation changes -and we end up with two same items in the array. It makes sense to squash these duplicates.
cc what do you think?
Why is aws csi driver provides PV with Seems to me this is not needed.
I believe aws csi driver recently changed to using kubernetes topology labels instead of vendor-specific topology labels. Note that the elements of a matchExpression are together. I'm not sure if that's the correct approach if the intention is to eventually get rid of the vendor-specific topology label. cc
I think the topology label is defined by the csi driver? Is it okay to get rid of the vendor specific label?
The intention was to put both labels on all nodes to maintain backwards compatibility with existing PVs. but provisioned PVs should only have one or the other, putting both labels on the PV is a bug: Now that we've released the bug and some PVs have both labels, it will be hard to remove the vendor specific label.
thanks for your follow up. After the provisioned PV is assigned to a node, removing the duplicate label would still be an issue ?
It depends on which label is going to be removed. I'd suggest to remove the in-tree one (), because we can be sure that nodes have the other label from registering the CSI driver.
+1
/triage accepted /assign
fixed by Closing


Question: } } <ins> // nodeInSameCondition returns true if all effective types ("Status" is true) equals; // otherwise, returns false. func nodeInSameCondition(old []v1.NodeCondition, cur []v1.NodeCondition) bool { if len(old) == 0 && len(cur) == 0 { return true } c1map := map[v1.NodeConditionType]v1.ConditionStatus{} for _, c := range old { if c.Status == v1.ConditionTrue { c1map[c.Type] = c.Status } } for _, c := range cur { if c.Status != v1.ConditionTrue { continue } if _, found := c1map[c.Type]; !found { return false } delete(c1map, c.Type) } return len(c1map) == 0 } </ins> func (dsc *DaemonSetsController) updateNode(old, cur interface{}) { oldNode := old.(*v1.Node) curNode := cur.(*v1.Node) <del> if reflect.DeepEqual(oldNode.Labels, curNode.Labels) && reflect.DeepEqual(oldNode.Spec.Taints, curNode.Spec.Taints) { // If node labels and taints didn't change, we can ignore this update. </del> <ins> if reflect.DeepEqual(oldNode.Labels, curNode.Labels) && reflect.DeepEqual(oldNode.Spec.Taints, curNode.Spec.Taints) && nodeInSameCondition(oldNode.Status.Conditions, curNode.Status.Conditions) { // If node labels, taints and condition didn't change, we can ignore this update. </ins> return } dsList, err := dsc.dsLister.List(labels.Everything()) if err != nil { glog.V(4).Infof("Error enqueueing daemon sets: %v", err)

deamonset would not schedule pod on nodes of OutofDisk state,on the other hand, in it's updateNode func, it return directly if there is no change in labels or taints of the node. That is, ds controller could not make sense when a node turn into OutofDisk state, or recover from it. Unless we restart the controller-manager component
/assign
if other conditions in func "nodeShouldRunDaemonPod" changed? say, a node does not have enough cpu or memory resource to create the pod of ds, later I delete pods on it, and it satisfy the request now, but the ds controller could not feel it
, thanks for your cases, they're reasonable to me; let me find solutions for them.


Question: ssc.podLister = podInformer.Lister() ssc.podListerSynced = podInformer.Informer().HasSynced <del> setInformer.Informer().AddEventHandlerWithResyncPeriod( </del> <ins> setInformer.Informer().AddEventHandler( </ins> cache.ResourceEventHandlerFuncs{ AddFunc: ssc.enqueueStatefulSet, UpdateFunc: func(old, cur interface{}) {

<!-- Please only use this template for submitting enhancement requests --/area controller-manager What would you like to be : Don't resync statefulset in statefulset controller. Why is this needed: There are 2000+ sts in my k8s cluster, thus when i create a new sts, it takes a lot of time to create it's pod, i found that every sts will cost 40ms to sync it, then 2000 sts will cost 80s, and there is also a statefulSetResyncPeriod to resync all sts, I don't know why there need to have statefulSetResyncPeriod to resync all sts? Can we not resync sts and just watch it's changes?
/sig api-machinery
/remove-sig api-machinery /sig apps /sig scalability
the resync pattern is followed by all controllers to account for bugs in watch mechanism or missed notification due to any other reason. I dont think the individual controllers resync period is configurable or we could increase it . Also i am not sure if the current scale limits of 2000 StatefulSets is supported FYI
I have three cluster, A has 20 sts, B has 1300 sts and C has 2300 sts. I set kube-controller-manager loglevel to 4 and find that the cost of processing every sts is 2ms,30ms,45ms. Can we record the processed object into a processed cache and every time we get a new object to precess, we can compare it with the item in the cache, if they are same, then we don't need to sync it.
The "resync pattern" doesn't help with watch bugs (resync is done purely from internal cache, so in hypothetical situation that we missed watch event, it won't help at all). Resync-period should basically be used only in controllers that also sync some external things (like load-balancers), and watching changes of those is impossible. Unless I'm not missing anything, resync period can be safely dropped in this case.
I think all the controllers have the same problem because of the sharedinformerfactory has a default resync period if we don't set it to 0 in the config file. Also if there are more sts, the consumption for every sts will be larger.
interesting, but the resync is only happening for k8s objects.

Why is watching changes on those impossible ? And when you say external things you mean loadbalancers in gcloud ? resync is only happening for k8s services corresponding to those loadbalancers , right ? Or may be you are saying that if a k8s services is linked to a gcloud loadbalancer, the resync enables the controller code to go and check gcloud if the loadbalancing is provisioned or has anything changed there ? We need to document this in somwhere much like the apiconventions doc
- I mean loadbalancer in GCP as an example (but can also be other load balancer, persistent volumes in cloud or things like that). Yes - we resync on k8s objects, but that that let's you sync those. As an example - imagine service of type LoadBalancer - if you're periodically syncing the k8s service object, this triggers the operation of syncing the load-balancer (and we have an ability to check if it exists, is healthy etc.). - hmm, I would need to double check it, but I thought the default resync period is pretty long...
The default resync period is 12~24 hours. It is long but the bad experience will still happen every day.