Question: return nil, false } <ins> // getPhase returns the phase of a pod given its container info. func getPhase(spec *api.PodSpec, info api.PodInfo) api.PodPhase { if info == nil { return api.PodPending } running := 0 waiting := 0 stopped := 0 failed := 0 succeeded := 0 unknown := 0 for _, container := range spec.Containers { if containerStatus, ok := info[container.Name]; ok { if containerStatus.State.Running != nil { running++ } else if containerStatus.State.Termination != nil { stopped++ if containerStatus.State.Termination.ExitCode == 0 { succeeded++ } else { failed++ } } else if containerStatus.State.Waiting != nil { waiting++ } else { unknown++ } } else { unknown++ } } switch { case waiting > 0: // One or more containers has not been started return api.PodPending case running > 0 && unknown == 0: // All containers have been started, and at least // one container is running return api.PodRunning case running == 0 && stopped > 0 && unknown == 0: // All containers are terminated if spec.RestartPolicy.Always != nil { // All containers are in the process of restarting return api.PodRunning } if stopped == succeeded { // RestartPolicy is not Always, and all // containers are terminated in success return api.PodSucceeded } if spec.RestartPolicy.Never != nil { // RestartPolicy is Never, and all containers are // terminated with at least one in failure return api.PodFailed } // RestartPolicy is OnFailure, and at least one in failure // and in the process of restarting return api.PodRunning default: return api.PodPending } } </ins> // GetPodStatus returns information from Docker about the containers in a pod func (kl *Kubelet) GetPodStatus(podFullName string, uid types.UID) (api.PodStatus, error) { <del> var manifest api.PodSpec </del> <ins> var spec api.PodSpec </ins> for _, pod := range kl.pods { if GetPodFullName(&pod) == podFullName { <del> manifest = pod.Spec </del> <ins> spec = pod.Spec </ins> break } } <del> info, err := dockertools.GetDockerPodInfo(kl.dockerClient, manifest, podFullName, uid) </del> <ins> info, err := dockertools.GetDockerPodInfo(kl.dockerClient, spec, podFullName, uid) </ins> <del> // TODO(dchen1107): Determine PodPhase here </del> var podStatus api.PodStatus <ins> podStatus.Phase = getPhase(&spec, info) netContainerInfo, found := info[dockertools.PodInfraContainerName] if found { podStatus.PodIP = netContainerInfo.PodIP } // TODO(dchen1107): Change Info to list from map </ins> podStatus.Info = info return podStatus, err

Kubelet now only reports ContainerStatus. PodStatus / PodCondition is observed at master side based on a set of ContainerStatus. There is no other way to get PodIP except net container for master side. This is very confusion to the end user, , , ... Now we are finalizing on v1beta3 API, including PodStatus & ContainerStatus. We can make net container completely invisible to the end users by finishing the following steps: 1) Expose network related errors at API level 2) Generate PodStaus directly from Kubelet
3) once PodStatus generated by kubelet, PodIP can be removed from ContainerStatus.
I think reporting PodStatus rather than just container status is most important. How hard is that?
Used to be hard since there are many dependencies when I first report Status back. Now it is totally feasible. I already started to write PRs for this. :-)
One more PR required to close this one.


Question: Args:    []string{"-c", "echo '" + content + "' > /mnt/index.html && chmod o+rX /mnt /mnt/index.html"}, VolumeMounts: []v1.VolumeMount{ { <del> Name:      config.Prefix + "-volume", </del> <ins> Name:      volMountName, </ins> MountPath: "/mnt", }, },

Failing Job: These tests started flaking recently (they were consistently green for a long time): Mounted volume expand[Slow] Should verify mounted devices can be resized Volume expand [Slow] Verify if editing PVC allows resize Subpath [Volume type: gcePDPVC] should support restarting containers using file as subpath [Slow] Subpath [Volume type: gcePDPVC] should fail for new directories when readOnly specified in the volumeSource Subpath [Volume type: gcePDPVC] should fail if subpath with backstepping is outside the volume [Slow] Volumes PD should be mountable with ext4 Affected runs: https://k8s- https://k8s- https://k8s- /kind bug /sig storage /priority important-soon /milestone v1.12 /cc - Could you PTAL or delegate? Thanks.
Volume expansion tests have been fixed by
/assign can you investigate the remaining failures?
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 4 days, the issue will be moved out of the v1.12 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Reopening this issue, as a bunch of storage-related e2es started flaking in our large-scale jobs recently: https://k8s- https://k8s- /reopen
I think the latest failures are caused by gce API rate limiting: /assign
It seems to only be failing on scalability test suites. I wonder if it's due to some different cidr configuration
/sig scalability
/kind failing-test
/status approved-for-milestone Friendly ping. This is affecting release-blocking scalability job.
hey what needs to be done here? by whom?
Could you comment on what needs to be done here and by whom?
can you guys take a look at the API quota issue?

This is happening consistently in the scale test environment
as Code Freeze is today, this needs to be of priority . Please triage! cc
It looks like the latest run on 9/1 has passed.
Marking closed as the fixes have been merged and the test runs are now passing /close
Closing this issue. <detailsIn response to : Marking closed as the fixes have been merged and the test runs are now passing /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func (util *ISCSIUtil) AttachDisk(b iscsiDiskMounter) error { var devicePath string <del> if b.iface == "default" { </del> <ins> var iscsiTransport string out, err := b.plugin.execCommand("iscsiadm", []string{"-m", "iface", "-I", b.iface, "-o", "show"}) if err != nil { glog.Errorf("iscsi: could not read iface %s error: %s", b.iface, string(out)) return err } iscsiTransport = extractTransportname(string(out)) if iscsiTransport == "" { glog.Errorf("iscsi: could not find transport name in iface %s", b.iface) return errors.New(fmt.Sprintf("Could not parse iface file for %s", b.iface)) } else if iscsiTransport == "tcp" { </ins> devicePath = strings.Join([]string{"/dev/disk/by-path/ip", b.portal, "iscsi", b.iqn, "lun", b.lun}, "-") } else { devicePath = strings.Join([]string{"/dev/disk/by-path/pci", "*", "ip", b.portal, "iscsi", b.iqn, "lun", b.lun}, "-") } <del> exist := waitForPathToExist(devicePath, 1, b.iface) </del> <ins> exist := waitForPathToExist(devicePath, 1, iscsiTransport) </ins> if exist == false { // discover iscsi target out, err := b.plugin.execCommand("iscsiadm", []string{"-m", "discovery", "-t", "sendtargets", "-p", b.portal, "-I", b.iface})

open-iscsi software initiators are currently broken for non- interfaces due to the fact that the devicePath (in ) is assumed to use the prefixing scheme for all interfaces not named . This causes waitForPathToExist to timeout after 10 seconds since the device path that it is waiting for is not the - prefixed path that is created, resulting in host logging such as: or K8S event logs such as: open-iscsi support for hardware transports was in the following commit: In this commit, software iSCSI support was assumed to be done through only the use of the default interface. According to the documentation at (Section 5.1.1) both software and offload/hardware interfaces are intended to be configured using named interface specifications and is only a backwards-compatibility vehicle. Configuration of cannot be done via the use of as this approach is specifically forbidden in the documentation: Leaving this here to help others associate various failure modes with this issue and possible resolution: In my case I am just learning and this is my first iSCSI deployment. My initial error was that I was receiving an message at discovery-time even though I had used to define a name. This approach failed because I didn't realize that iscsid needed to be bounced before it would pick up changes to that file. I had defined a custom specification and was using that to accomplish the task of setting my initiator name. I was able to unwind this mess and return to the use of once I read the code and understood what it was attempting to do.
i'm marking this as a feature request against the iscsi volume driver. RFE: support iscsi interfaces named something besides 'default'.
In theory the fix should be easy, just move the path check from iface name to transport_name. I will take a look asap, unfortunately I am travelling till monday
Sorry about the previous comment, I was logged in to the wrong account. In theory the fix should be easy, just move the path check from iface name to transport_name. I will take a look asap, unfortunately I am travelling till monday. can you share your iface file ? It will help me validate
I no longer have the file precisely as it was. However, I did have some version of what I had used committed into my VCS. While not 100% sure this is what I was using while encountering the failure it does looks to be about what I'd expect:


Question: InitContainers: []v1.Container{ { SecurityContext: &v1.SecurityContext{ <del> RunAsUser: utilptr.Int64Ptr(999), </del> <ins> RunAsUser: ptr.To[int64](999), </ins> }, }, }, Containers: []v1.Container{ { SecurityContext: &v1.SecurityContext{ <del> RunAsUser: utilptr.Int64Ptr(1000), </del> <ins> RunAsUser: ptr.To[int64](1000), </ins> }, }, { SecurityContext: &v1.SecurityContext{ <del> RunAsUser: utilptr.Int64Ptr(1000), </del> <ins> RunAsUser: ptr.To[int64](1000), }, }, }, EphemeralContainers: []v1.EphemeralContainer{ { EphemeralContainerCommon: v1.EphemeralContainerCommon{ SecurityContext: &v1.SecurityContext{ RunAsUser: ptr.To[int64](1001), }, </ins> }, }, },

projected volumes exclude ephemeral containers in calculating fs user because "SecurityContext is not allowed" : This is no longer technically correct -- ephemeral containers are allowed to specify security context -- but it's a no-op because is guaranteed to be empty at pod creation when fs user is calculated. Still, the comment is wrong so let's clean it up to avoid confusion. /kind cleanup
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /assign
/priority backlog
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten


Question: concurrent-serviceaccount-token-syncs concurrent-service-syncs config-map <del> config-map-data </del> config-map-namespace config-sync-period configure-cloud-routes

kubectl edit is not working on HEAD. It's working on 1.5.


Question: }, Spec: experimental.DeploymentSpec{ Replicas:       1, <del> Selector:       podLabels, </del> <ins> Selector:       deploymentPodLabels, </ins> UniqueLabelKey: "deployment.kubernetes.io/podTemplateHash", Template: &api.PodTemplateSpec{ ObjectMeta: api.ObjectMeta{ <del> Labels: podLabels, </del> <ins> Labels: deploymentPodLabels, </ins> }, Spec: api.PodSpec{ Containers: []api.Container{

cc
Thanks for flagging it! Its not flaky, it is broken as we did not enable deployments for that test suite. Sent to fix that.


Question: } name := env[:pos] value := env[pos+1:] <del> if len(name) == 0 || len(value) == 0 { </del> <ins> if len(name) == 0 { </ins> return nil, fmt.Errorf("invalid env: %v", env) } if len(validation.IsCIdentifier(name)) != 0 {

It should be possible to pass empty environment variables to pods run by env (to clear an env var set inside the image, for example). is valid syntax for "set env var FOO to empty string".


Question: mutex sync.Mutex <del> // We must cache because otherwise there is a race condition, // where we assign a device mapping and then get a second request before we attach the volume deviceMappings map[mountDevice]string </del> <ins> // We keep an active list of devices we have assigned but not yet // attached, to avoid a race condition where we assign a device mapping // and then get a second request before we attach the volume attaching map[mountDevice]string </ins> } // newAWSInstance creates a new awsInstance object

Environment: CoreOS 815.0.0, k8s 1.1.0 () It happens to me quite often that a volume marked as "available" on AWS remains in a "detached" state according to the kubelet: Deleting the pod and recreating it eventually succeeds.
I have some fixes in progress that should fix this
/cc
I can reliably reproduce similar issue, I am not sure it's the same: create a AWS EBS, let's say it has ID Create a pod that uses this EBS: Create the pod. It succeeds eventually: Delete the pod Create the pod again. It will be forever IMO, it's caused by not removing the detached volume from internal map of attached volumes . Subsequent will find it there and it won't even try to attach it again. Patch is available in referenced PR.
can you please confirm that your bugs is the same as I can reproduce? Or do you see different bug, only with similar symptoms?
It is
Was this fixed by ?
I think you referenced the wrong issue / PR, didn't you? As far as I know this is not fixed yet.
Yes, I meant to link . mentioned there that it might fix this issue.
Oh right! It made it to v1.2.0-alpha.4
did not fix this issue (but fixed the DeviceMapping cache invalidation)
I think, what is happening is that you run into an error or timeout while waiting for the disk operation. That leaves the devicecache yet again in an inconsistent state. I have a fix here that just discards the cache on any error. Additionally, it is totally possible that there is concurrent operations which lead to an error and bork the cache. The fix: Unfortunately, this is in our tests still not 100% rock solid. Especially in timeout cases. I'm missing the capability to properly test this to engage in a bigger refactoring of this whole issue. A workaround to actually fixing it properly would be to add a TTL to the cache, so that it eventually starts working again.
Just for information for people who might use the GitHub search, as of Kubernetes 1.1.7 the errors logged by the look like: Deleting the pod or restarting doesn't help, the instance keeps failing attaching the EBS with the same errors if the pod get rescheduled on the same host.

thanks for the fix!
Oh... the same host. I bet I messed up that case... I'm going to move into 1.2 so I take another look at this!
What's the status on this?
I just don't think this is fixable with the volume cache we have. I proposed a PR to remove the cache, and replace it with a map of active attachment operations in progress. A little less efficient, but a lot more correct (presuming passes testing).
How to check what K8 version has this fix?
Is there any progress on this issue? I'm running K8s version v1.5.2, and I'm having similar issues. awsElasticBlockStore volumes are virtually unusable due to one of three errors I'm getting: My volume claim is ReadWriteOnce, so this next error confuses me: And finally: [SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected.] Unable to mount volumes for pod "staging-connect-mysql-<id(<id)": timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql] Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql]
Also having the same issue as ^ Should we be using PV+VPC instead of claiming the EBS volume directly from the deployment spec?


Question: defer t.c.CoreV1().ConfigMaps(t.ns).Delete(t.name, nil) t.createUtilPodLabel("e2e-dns-configmap") defer t.deleteUtilPod() <ins> originalConfigMapData := t.fetchDNSConfigMapData() defer t.restoreDNSConfigMap(originalConfigMapData) </ins> t.validate() <del> t.labels = []string{"abc", "ghi"} valid1 := map[string]string{"federations": t.labels[0] + "=def"} valid1m := map[string]string{t.labels[0]: "def"} valid2 := map[string]string{"federations": t.labels[1] + "=xyz"} valid2m := map[string]string{t.labels[1]: "xyz"} invalid := map[string]string{"federations": "invalid.map=xyz"} </del> <ins> if t.name == "coredns" { t.labels = []string{"abc", "ghi"} defaultConfig := map[string]string{ "Corefile": `.:53 { kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } }`} </ins> <del> By("empty -> valid1") t.setConfigMap(&v1.ConfigMap{Data: valid1}, valid1m, true) t.validate() </del> <ins> valid1 := map[string]string{ "Corefile": `.:53 { kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } federation cluster.local { abc def.com } proxy . /etc/resolv.conf }`} valid1m := map[string]string{t.labels[0]: "def.com"} </ins> <del> By("valid1 -> valid2") t.setConfigMap(&v1.ConfigMap{Data: valid2}, valid2m, true) t.validate() </del> <ins> valid2 := map[string]string{ "Corefile": `:53 { kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } federation cluster.local { ghi xyz.com } proxy . /etc/resolv.conf }`} valid2m := map[string]string{t.labels[1]: "xyz.com"} </ins> <del> By("valid2 -> invalid") t.setConfigMap(&v1.ConfigMap{Data: invalid}, nil, false) t.validate() </del> <ins> By("default -> valid1") t.setConfigMap(&v1.ConfigMap{Data: valid1}, valid1m, true) t.deleteCoreDNSPods() t.validate() </ins> <del> By("invalid -> valid1") t.setConfigMap(&v1.ConfigMap{Data: valid1}, valid1m, true) t.validate() </del> <ins> By("valid1 -> valid2") t.setConfigMap(&v1.ConfigMap{Data: valid2}, valid2m, true) t.deleteCoreDNSPods() t.validate() </ins> <del> By("valid1 -> deleted") t.deleteConfigMap() t.validate() </del> <ins> By("valid2 -> default") t.setConfigMap(&v1.ConfigMap{Data: defaultConfig}, nil, false) t.deleteCoreDNSPods() t.validate() </ins> <del> By("deleted -> invalid") t.setConfigMap(&v1.ConfigMap{Data: invalid}, nil, false) t.validate() </del> <ins> } else { t.labels = []string{"abc", "ghi"} valid1 := map[string]string{"federations": t.labels[0] + "=def"} valid1m := map[string]string{t.labels[0]: "def"} valid2 := map[string]string{"federations": t.labels[1] + "=xyz"} valid2m := map[string]string{t.labels[1]: "xyz"} invalid := map[string]string{"federations": "invalid.map=xyz"} By("empty -> valid1") t.setConfigMap(&v1.ConfigMap{Data: valid1}, valid1m, true) t.validate() By("valid1 -> valid2") t.setConfigMap(&v1.ConfigMap{Data: valid2}, valid2m, true) t.validate() By("valid2 -> invalid") t.setConfigMap(&v1.ConfigMap{Data: invalid}, nil, false) t.validate() By("invalid -> valid1") t.setConfigMap(&v1.ConfigMap{Data: valid1}, valid1m, true) t.validate() By("valid1 -> deleted") t.deleteConfigMap() t.validate() By("deleted -> invalid") t.setConfigMap(&v1.ConfigMap{Data: invalid}, nil, false) t.validate() } </ins> } func (t *dnsFederationsConfigMapTest) validate() {

/kind failing-test /priority critical-urgent /sig network /milestone v1.12 /cc
ping
Also failures on
TLS errors connection refused 503 service unavailable especially around metrics and healthz in ways that make me suspect things didn't start correctly. Or perhaps API mismatch? /cc 've looked at some similar failures in the past. Any chance you might give a look over this and ?
in both and here looks like api mismatch?
Test output shows no DNS server can be reached, but logs say coredns pods are running (https://k8s-): cc
The CoreDNS configmap is incorrectly configured and hence the test fails. I'm working on the fix.
/assign
, , Does it make sense to keep that e2e test ?, knowing that this test is specific for "Federation" and that federation has now merged into "multicluster" and removed all e2e tests about "Federation + Serial" (formerly "ci-federation-e2e-gce-serial") : can you say if that use case is still supported in kubernetes v1.12 ?
I can say that Federation v1 has not been exercised since v1.10. I think a good question for the Federation v2 folks is not only whether they still need the tests, but also whether they need the configmap support specific to what Federation v1 used -- I'll defer to
If ya'll decide the use case is no longer valid, please open an issue / remove the test once 1.13 release opens.
tests are failing
/reopen /cc
Reopening this issue. <detailsIn response to : /reopen /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
More failures:
Looks like this is failing since Kube-dns is now enabled.
Since we'd previously discussed this not being a relevant test case, and it seems less appropriate even if it's this coupled to the dns implementation and we've got the kube-dns to coredns transition happening in the next release most likely....Can we just disable/remove the test?
Opened to disable this test.
Thank you!


Question: return selector.Matches(podLabelsSelector) }, nil } <ins> // ReplicaSetIsInactive returns a condition that will be true when a replica set is inactive ie. // it has zero running replicas. func ReplicaSetIsInactive(c unversionedextensions.ExtensionsInterface, replicaSet *extensions.ReplicaSet) wait.ConditionFunc { // If we're given a ReplicaSet where the status lags the spec, it either means that the // ReplicaSet is stale, or that the ReplicaSet manager hasn't noticed the update yet. // Polling status.Replicas is not safe in the latter case. desiredGeneration := replicaSet.Generation return func() (bool, error) { rs, err := c.ReplicaSets(replicaSet.Namespace).Get(replicaSet.Name) if err != nil { return false, err } return rs.Status.ObservedGeneration >= desiredGeneration && rs.Spec.Replicas == 0 && rs.Status.Replicas == 0 && rs.Status.FullyLabeledReplicas == 0, nil } } </ins>

The scenario is a single replica with an attached EBS. The create seems to happen right after the delete not waiting for the pod to be terminated. This results in the two kubelets fighting over the EBS which generally results in AWS becoming unhappy meaning the EBS has to be force detached. From the docs I would have thought deployments would wait until the pod is actually killed. Is this a bug or am i misinterpreting the docs?
As a work around, what happens if you use a rolling update with MaxSurge=0 and MaxUnavailable=1?
Just assigned for triage, not to fix the issue. Can clear your assignment once this is triaged.
Deployments with size of 1 should be the same (both in that it's really a Recreate and it has the same problem even though it only matters for Recreate), larger deployments should be rolling.
It's because Deployment scales old RS down to 0 (=0, and RS controller will observe this change later and then kill its pods) and then scales up the new one. So it's possible there's a race here.
Opened that partially fixes this.
Why it's closed? It's not fixed: Recreate still doesn't wait for termination.
what version are you using?
1.6.2-00 from deb package. I'm changing the Deployment to use different configmap. K8s recreates pod, but the new one can't listen on ports that are probably used by the old one (using ). Both are appearing in at the same time (old one is 'Terminating').
This regressed in 1.6 but was fixed again in I don't think it was cherry-picked back to 1.6 though and the cherry-pick is not clean. are you interested in backporting it to the release-1.6 branch?
the cherry-pick isn't clean mostly because of comments. Don't have time to actually build and test the PR though.
I think this is still an issue:
I cannot reproduce it anymore, can you provide a reproducer?
Not easily at the moment -- we ended up changing our pods to handle multiple running simultaneously gracefully.
Hi All, I'm still seeing this on the following version: I am using recreate and can see my current pod go into a terminating state and my new pod come up.

It is my understanding that the new pod should not come up until the old pod is terminated completely.
Seeing this on 1.14.3 too - AWS hardware went bad and stuff was in terminating (but still running) and our stateful singleton was brought back up alongside
We see the same problem with 1.14.3 where the old pod is Terminating and the new pod is in ContainerCreating even if we set the Got problem with Volume is already used by pod(s)...
The easiest way I've found to reproduce this is by using bash interactively - it swallows SIGTERM giving a bigger window to observe the behavior. One thing to note, the behavior is different with an explicit delete and one caused by a changing annotation. When changing an annotation, the new pod won't be scheduled until the old one is completely gone, but an explicit delete will create the new pod right away. That said, I don't know how it will behave with other events, say a node going away or being drained. Here is a way to reproduce it. Delete the pod () and watch the new one spin up right away.
recreate deployments wait for the termination of an old version only during a deployment upgrade. The deployment api has been designed with availability in mind and cannot ensure consistency during node upgrades (use PDBs), unexpected node shutdowns, or any other reason that could cause a pod to be recreated (apart from deployment upgrades). If your application has a hard requirement on consistency (eg. database), use . can you please provide a reproducer? Thanks.
Same issue here with version v1.17.6: Deployment with "Recreate" strategy: If I try to delete a pod, the old pod is still Terminating, the new pod has already get initialized and Running:
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
see the comment right above you "recreate deployments wait for the termination of an old version only during a deployment upgrade.

The deployment api has been designed with availability in mind and cannot ensure consistency during node upgrades (use PDBs), unexpected node shutdowns, or any other reason that could cause a pod to be recreated (apart from deployment upgrades). If your application has a hard requirement on consistency (eg. database), use statefulsets."


Question: return fmt.Errorf("Failed to ensure that %s chain %s jumps to %s: %v", tc.table, tc.chain, kubeHostportsChain, err) } } <del> // Need to SNAT traffic from localhost args = []string{"-m", "comment", "--comment", "SNAT for localhost access to hostports", "-o", natInterfaceName, "-s", "127.0.0.0/8", "-j", "MASQUERADE"} if _, err := iptables.EnsureRule(utiliptables.Append, utiliptables.TableNAT, utiliptables.ChainPostrouting, args...); err != nil { return fmt.Errorf("Failed to ensure that %s chain %s jumps to MASQUERADE: %v", utiliptables.TableNAT, utiliptables.ChainPostrouting, err) </del> <ins> if natInterfaceName != "" && natInterfaceName != "lo" { // Need to SNAT traffic from localhost args = []string{"-m", "comment", "--comment", "SNAT for localhost access to hostports", "-o", natInterfaceName, "-s", "127.0.0.0/8", "-j", "MASQUERADE"} if _, err := iptables.EnsureRule(utiliptables.Append, utiliptables.TableNAT, utiliptables.ChainPostrouting, args...); err != nil { return fmt.Errorf("Failed to ensure that %s chain %s jumps to MASQUERADE: %v", utiliptables.TableNAT, utiliptables.ChainPostrouting, err) } </ins> } return nil }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In Ubuntu 18.04 DNS queries are being served via daemon which is listening on and serves queries only from . The only line in is , and this file is managed by systemd. Once any pod with notation appears on the host, fires the iptables rule in chain of table: -A POSTROUTING -s 127.0.0.0/8 -o lo -m comment --comment "SNAT for localhost access to hostports" -j MASQUERADE See commit and lines for details. Therefore all queries to goes not from , but from interface with default route due to masquerading, and rejects all of these requests with systemd-resolved[]: Got packet on unexpected IP range, refusing. How to reproduce it (as minimally and precisely as possible): Just install Kubernetes with CNI on Ubuntu-18.04 (with kubeadm, kubespray, chef-kubernetes, whatever) and run any pod with . Voila! You just lost DNS resolving. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Digital Ocean (but reproduceable on any configuration) OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: (but reproduceable with any install tool) Others: CNI: 0.6.0 CNI plugins: 0.7.1 CNI daemon: weave /sig network
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Hi, is anyone managed to figure a workaround for this?
Don't use Ubuntu 18.04?
Please if you have nothing useful to add, just move along. The workaround that I have settled on, is to run the following command on your Ubuntu 18.04 system during your bootstrapping: This ^^ solves the issue for me.
Can you explain what that means? Thank you!

Adding an SNAT rule above the rule generated by kubelet fixes it (replace the with whatever the slot is above the offending rule for you): I'm going to look at upstreaming this fix.
Thanks for your solution! Is it possible to fix on systemd side?
reopen if needed
but it is still broken, why did you close issue?
I have the same issue on Kubernetes version 1.14.2 on Ubuntu 18.04
Fixed in
Bug still present in kubernetes 1.16.0
I don't know if people still encounter this issue. For kops users, the solution I've found is to add this to cluster spec: This will actually tell kubelets to use the systemd insted of the default one in . For the users not using kops, you can just set in the kublet options.
I'm still having it on Ubuntu 19.10 running latest 1.6.2 minikube (aka k8s v1.17.0) but with "cri-o" as container engine which enables CNI by default. Since cri-o is very slow with releasing .deb packages - it is 1.15 which may be still impacted by this issue. Adding to solves issue - but it does not solve issue described in here. Temporarily I solve it by running: after first DaemonSet starts first Pod.
Update: After updating cri-o from 1.15 to 1.16 the problem vanished.


Question: closable := &closableConn{Conn: conn} <del> // Start tracking the connection d.mu.Lock() d.conns[closable] = struct{}{} d.mu.Unlock() </del> // When the connection is closed, remove it from the map. This will // be no-op if the connection isn't in the map, e.g. if CloseAll() // is called.

What happened: my kubelet restart unexpectedly,i find log info has more "Failed to list" info and has fatal error: go of nil func value,and then kubelet restart Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"logging"/"elasticsearch-token-99wc2": Failed to list v1.Secret: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] failed 5 attempts to update node lease, will retry after 10s Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"logging"/"default-token-6wh7n": Failed to list v1.Secret: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"monitoring"/"prometheus-rules": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"logging"/"service-registry": Failed to list v1.Secret: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"monitoring"/"prometheus-center": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:30 mgt15 kubelet[6914]: E0202 17:46:30. 6914 ] Failed to list v1beta1.RuntimeClass: Get https://vip- Feb 02 17:46:31 mgt15 kubelet[6914]: E0202 17:46:30. 6914 ] object-"logging"/"telegraf-elasticsearch": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:31 mgt15 kubelet[6914]: E0202 17:46:30.

6914 ] object-"monitoring"/"prometheus-center": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:32 mgt15 kubelet[6914]: fatal error: go of nil func value Feb 02 17:46:32 mgt15 kubelet[6914]: goroutine 357 [running]: Feb 02 17:46:32 mgt15 kubelet[6914]: (closableConn).Close(0x4002193f00, 0x0, 0x0) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- Feb 02 17:46:32 mgt15 kubelet[6914]: (Dialer).CloseAll(0x4000bbaea0) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- Feb 02 17:46:32 mgt15 kubelet[6914]: (Kubelet).updateNodeStatus(0x40010ce000, 0x54d0978, 0x40010ce648) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- +0xf0 Feb 02 17:46:32 mgt15 kubelet[6914]: (Kubelet).syncNodeStatus(0x40010ce000) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- What you expected to happen: kubelet do not restart How to reproduce it (as minimally and precisely as possible): when have more info of "Failed to list" Environment: Kubernetes version (use ):v1.14.3 OS (e.g: ):Ubuntu 18.04.3 LTS Kernel (e.g.

):4.15.0-55-generic Network plugin and version (if this is a network-related bug):calico3.8.2
/sig node /wg node /committee node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig node /wg node /committee node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fix is in
/sig auth
/assign
thankswhat version is this bug been fixed?
The fix is not yet merged, but I plan to backport the fix to 1.15.x, 1.16.x, 1.17.x (the exact patch versions depend on when the backports are merged)


Question: nonstatics+=($binary) fi done if [[ "${#statics[@]}" != 0 ]]; then kube::golang::fallback_if_stdlib_not_installable; fi

disabled PPC builds due to the following error: Additional comments here
cc
The issue on PowerPC is both for cross and native-builds. The issue appears to be with the toolchain (golang) and how the linking is being done. The object sections have grown too large to link. We are looking at it cc
Have you tried to upgrade to a newer version? It might or might not help... You might also try to look at what in the gcc linker differs from for example, where it doesn't fail
We are looking into this and it might take some more time. In the meanwhile is it possible to remove hyperkube from the ppc64le builds? All other binaries are building perfectly fine. We can resume the hyperkube for ppc64le build as soon as this issue resolves. cc:
Ah, this is only partially fixed. Reopening...
The error occurs because the hyperkube binary is very large and when building the binary, golang is generating a single go.o file with a single text section that is larger than 2^26 on ppc64le. When there is a call instruction within that section where the call target offset is beyond what can fit in the instruction the relocation truncation error will be generated by the GNU linker. When there are multiple, smaller text sections sent to the GNU linker for linking, it can generate appropriate call instructions even the call target is beyond that limit. I'm working on a fix to do that but hitting other issues as a result which I'm working through. I suspect that this same error will be hit at some point on arm64 unless their linker has another way to handle this situation. On ppc64le we are very close to the limit and from build to build it varies so that some days it will build and others it won't.
Interesting. How does amd64 handle it?
I don't know for sure how arm64 handles it because I don't have access to a machine that is set up to build for arm64. I suspect their text section size is not as large as it is on ppc64le so they haven't hit the limit yet.
Ok, you may easily see what's going on for arm64 by investigating what does in kube-cross.

But I asked about what amd64 (x86_64) does to prevent this :) (I'm assuming this will never happen to amd64)
Sorry I misread your question about amd64... The call instruction in amd64 has a 32 bit field instead of a 24 bit field in ppc64le for the call target address. That means they wouldn't hit this error when linking hyperkube until its text section was larger than 2^32. If you search on "relocation truncated to fit" you will see that this error has happened in the past on x86 with other programming languages.
Once a fix is ready, how does it need to be provided?
I saw https://go- probably won't make 1.7, and that's sad. We would just upgrade golang version I think, not really sure what you're asking for
I assume you only use official golang versions or those provided with distros for your builds? Are you doing all your builds as cross builds from a golang on an x86?
Official versions, yes. We're using the docker image. All official builds are built on amd64 hosts.
Where are you getting this docker image from? Will you be moving to golang:1.7 when it is released?
We're using the official images, and will pick up as soon as it is released for testing I think. Then if the testing and scalability teams think it's safe to use, we'll switch to 1.7 BTW, where is your patch for this? Can you link to it?
is also broken right now - see .
Hi, I'm sorry I missed your last question to me. Here is the CL for my patch for the PPC64 relocations truncation error. https://go- The error with kubemark is also size related but somewhat different because that one is being built as static. Just curious, why are some built static and some non-static?
just to tie everything together, disabled linux/ppc64le compilation entirely. hopefully by go1.8 we can reenable it.
Well, we've hit 1.8 I think with Want to link to any other relevant issues to kinda tie this one together, and close this if it seems appropriate?
yeah, ppc64le builds have been re-enabled for k8s 1.6.

The issue on PowerPC is both for cross and native builds. The issue appears to be with the toolchain (golang) and how the linking is being done. The object sections have grown too large to link. We are looking at it cc
this issue is quite old. Can you provide more context to your comment? Specifically, more detail to help me reproduce. I was unable to find anything related to ppc64le on .


Question: gotBinding = action.(clienttesting.CreateAction).GetObject().(*v1.Binding) return true, gotBinding, item.injectBindError }) <del> fwk, err := st.NewFramework([]st.RegisterPluginFunc{ </del> <ins> registerPluginFuncs := append(item.registerPluginFuncs, </ins> st.RegisterQueueSortPlugin(queuesort.Name, queuesort.New), st.RegisterBindPlugin(defaultbinder.Name, defaultbinder.New), <del> }, frameworkruntime.WithClientSet(client)) </del> <ins> ) fwk, err := st.NewFramework(registerPluginFuncs, frameworkruntime.WithClientSet(client)) </ins> if err != nil { t.Fatal(err) }

This is a bug found in The scheduler fails to schedule the pod on the first try due to PV is out of sync. This may happen because the PV/PVC may be modified before the scheduler binder assumes them. However, the scheduler skips the pod update event and never try to scheduler the pod again: This is because when the reserve plugins fails, we didn't try to forget the pod. After the is called, the pod is in the state.
/sig scheduling /sig storage /assign /priority critical-urgent
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig scheduling /sig storage /assign /priority critical-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, }, { <del> description: "pod specified a toleration for taint `notReady:NoExecute`, expect add toleration for `unreachable:NoExecute`", </del> <ins> description: "pod specified a toleration for taint `not-ready:NoExecute`, expect add toleration for `unreachable:NoExecute`", </ins> requestedPod: api.Pod{ Spec: api.PodSpec{ Tolerations: []api.Toleration{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When discussing issue in , I noticed: although is using key , it has actually been changed to in for quite a while. I am thinking we may need to fix this inconsistency in 1.9, by moving out of alpha. Anything else we need to know?: Task tracker: [x] update [x] update kubernetes doc
/sig scheduling


Question: wantMetricStrings := []string{ `apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds{apiserver_id_hash="sha256:3c607df3b2bf22c9d9f01d5314b4bbf411c48ef43ff44ff29b1d55b41367c795",status="success"} FP`, `apiserver_encryption_config_controller_automatic_reload_success_total{apiserver_id_hash="sha256:3c607df3b2bf22c9d9f01d5314b4bbf411c48ef43ff44ff29b1d55b41367c795"} 2`, <ins> `apiserver_encryption_config_controller_automatic_reloads_total{apiserver_id_hash="sha256:3c607df3b2bf22c9d9f01d5314b4bbf411c48ef43ff44ff29b1d55b41367c795",status="success"} 2`, </ins> } test.secret, err = test.createSecret(testSecret, testNamespace)

Change encryption config controller reload metrics to be consistent with other places (Authn, Authz) xref: Combine and into a single metric with with an enum value of and . /sig auth /triage accepted
cc
We probably should keep the existing metrics for some time to allow folks to migrate to the new one.
Ah I was thinking about this as well! Looking at "may be deleted at any time", I was leaning towards just removing it. If the common approach is to keep it for a release before removing, we could do that. I believe we can set a for the 2 metrics we want to remove.
I don't have a good sense of metrics stability because they tend to stay alpha forever and folks don't go back to update them. Since the associated feature set is GA, I think we should be conservative with the changes we make. Perhaps SIG instrumentation folks can provide some guidance here?
As part of the PR, I was going to ask when is the right time to graduate these metrics do you have thoughts on to take while deleting an alpha metric metrics should be graduated from alpha to beta and stable?


Question: rbacv1 "k8s.io/api/rbac/v1" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apiserver/pkg/authentication/user" </ins> "k8s.io/apiserver/pkg/authorization/authorizer" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/component-base/featuregate"

What happened: I checked if kubelet () has permissions for with SelfsubjectAccessReviews and I saw that it has. But when I checked its permissions with SelfsubjectRulesReviews it didn't show it. What you expected to happen: I expected that the API SelfsubjectRulesReviews will show that has permssions for . How to reproduce it (as minimally and precisely as possible): You can check it on the master or the worker. Test SelfsubjectAccessReviews: You will get answer that it has permissions: Test SelfsubjectRulesReviews: It won't show the permissions. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm /area/api /kind/bug /sig api-machinery
/sig api-machinery
This is a more clean display that show the bug: I think the problem is related to the fact that the file contain the kubelet certificate which is part of the group :
/sig auth
What is the SelfSubjectRulesReview response from the server ()? Note that not all authorizers support enumerating permissions, and the API object status includes an attribute which explicitly indicates the result is incomplete. See
It has Details:
I noticed that it also doesn't show that it has permissions to view and . It probably related to the Node authorizer: Because I assume that these rules doesn't appear in RBAC.
The node authorizer does not enumerate permissions, so it does not contribute to SelfSubjectRulesReview results
fixes setting correctly when the API is requested with a kubelet's credentials


Question: if resp.Url == "" { errorMessage := "URL is not set" <del> klog.Errorf("Exec failed: %s", errorMessage) </del> <ins> klog.Errorf("Attach failed: %s", errorMessage) </ins> return nil, errors.New(errorMessage) } return resp, nil

What happened: In , there is mismatch between method name and the method in logs. e.g. RemoteRuntimeService#Attach has the following: What you expected to happen: The method names should be consistent with what appear in the logs. Otherwise people would be confused when troubleshooting. /sig node
could i try to fix this?Thanks
Yes. fixes method name for ContainerStats. You can cover the rest. thanks
/assign


Question: deps = [ "//staging/src/k8s.io/cli-runtime/pkg/genericclioptions:go_default_library", "//staging/src/k8s.io/kubectl/pkg/cmd/util:go_default_library", <ins> "//vendor/github.com/spf13/cobra:go_default_library", </ins> ], )

What happened: I am using a command like below in GitHub actions: I get the following error: What you expected to happen: Since I am using flag, it should work without requiring a valid kubeconfig file. How to reproduce it (as minimally and precisely as possible): Just delete your local kubeconfig file and try the above command. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
/assign /kind regression
This likely breaks some scripted usage of kubectl
This is related to /assign
Or
Yes, sorry I mentioned you in this issue. Because my first thought is this brings and which will check the config file existence but in fact that's not the root cause. Actually the root cause This checks config has the server or not. And with this , this function will return error when there's no local config file. I think there might be two solutions? the similar logic like to every , , ..... when or is set. Seems not a good idea. reverse this to have the default server which is better than ? By the way, is equals ? Since This works only for .
The plan I agreed on with Jordan is that I will revert since that seems to be the main culprit behind all these problems. I'll add a test ensuring this works as required. This will allow us to backport the fix to 1.18. For 1.19 I'll put together a proper fix supporting and flags.


Question: framework.Failf("%v", err) } <del> customServiceShortName := "ksvc" crdSvc, err := setupCRDWithShortName(f, schemaCustomService, "service", customServiceShortName, "v1alpha1") if err != nil { framework.Failf("%v", err) } </del> meta := fmt.Sprintf(metaPattern, crd.Crd.Spec.Names.Kind, crd.Crd.Spec.Group, crd.Crd.Spec.Versions[0].Name, "test-foo") ns := fmt.Sprintf("--namespace=%v", f.Namespace.Name)

Which jobs are failing: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) Which test(s) are failing: [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema [Conformance] Since when has it been failing: 04-26 05:02 PDT Testgrid link: Reason for failure: Apr 26 12:28:01.219: FAIL: failed to explain error running /workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://35.185.126.243 -- explain Command stdout: stderr: the server doesn't have a resource type "ksvc" error: exit status 1 Full Stack Trace () +0x1545 (0xc002ca6100) +0x324 (0xc002ca6100) +0x2b testing.tRunner(0xc002ca6100, 0x4b8d060) +0xc9 created by testing.(T).Run +0x350 [AfterEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] ï¿½[1mSTEPï¿½[0m: Collecting events from namespace "crd-publish-openapi-4377". ï¿½[1mSTEPï¿½[0m: Found 0 events. Apr 26 12:28:01.314: INFO: POD NODE PHASE GRACE CONDITIONS Apr 26 12:28:01.314: INFO: Apr 26 12:28:02.899: INFO: skipping dumping cluster info - cluster too large Apr 26 12:28:02.899: INFO: Waiting up to 3m0s for all (but 50) nodes to be ready ï¿½[1mSTEPï¿½[0m: Destroying namespace "crd-publish-openapi-4377" for this suite. Anything else we need to know: /cc /cc /milestone v1.19 /priority critical-urgent
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

<detailsIn response to : Which jobs are failing: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema [Conformance] 04-26 05:02 PDT Apr 26 12:28:01.219: FAIL: failed to explain error running /workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://35.185.126.243 -- explain Command stdout: the server doesn't have a resource type "ksvc" exit status 1 () (0xc002ca6100) (0xc002ca6100) testing.tRunner(0xc002ca6100, 0x4b8d060) created by testing.(T).Run [AfterEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] ï¿½[1mSTEPï¿½[0m: Collecting events from namespace "crd-publish-openapi-4377". ï¿½[1mSTEPï¿½[0m: Found 0 events. Apr 26 12:28:01.314: INFO: POD NODE PHASE GRACE CONDITIONS Apr 26 12:28:01.314: INFO: Apr 26 12:28:02.899: INFO: skipping dumping cluster info - cluster too large Apr 26 12:28:02.899: INFO: Waiting up to 3m0s for all (but 50) nodes to be ready ï¿½[1mSTEPï¿½[0m: Destroying namespace "crd-publish-openapi-4377" for this suite. /cc /milestone v1.19 </details
/milestone v1.19 Thanks for opening this! I'll get you on milestone maintainers soon :) Also, was taking a look at this as well and noticed this test was recently :
As noted in , this modified an existing conformance test and made it flaky. Please move the new test out of the conformance test and mark it as flaky until this issue has been resolved.

Cc for flake introduced in
It is a bit weird that I cannot reproduce this problem on my local kind cluster by running even after 20 times I suspect this is a cache issue since I once created a crd without short name and forgot to delete it in the test.
/retitle [Flaky Test]: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) /kind flake /remove-kind failing-test
/assign
I can reproduce this problem by the following steps now. It looks like that this problem has something to do with discovery cache. <pre$ cat <summary<detailsapiVersion: kind: CustomResourceDefinition metadata: name: spec: conversion: strategy: None group: crd-publish-openapi-test- names: kind: Service listKind: ServiceList plural: services singular: service shortNames: ["ksvc"] scope: Namespaced versions: name: v1alpha1 schema: openAPIV3Schema: properties: spec: description: Specification of CustomService properties: dummy: description: Dummy property. type: string type: object type: object served: true storage: true </details</summary$ kubectl get crd No resources found in default namespace. $ rm -r $ kubectl explain the server doesn't have a resource type "ksvc" $ kubectl create -f created $ kubectl explain the server doesn't have a resource type "ksvc" $ kubectl explain <summary<detailsKIND: Service VERSION: crd-publish-openapi-test- RESOURCE: spec <ObjectDESCRIPTION: Specification of CustomService FIELDS: dummy <stringDummy property. </details</summary</pre
looks like
/assign


Question: } var defaultParameterEncoder runtime.ParameterCodec = parameterCodec{} <ins> type versionedParameterEncoderWithV1Fallback struct{} func (versionedParameterEncoderWithV1Fallback) EncodeParameters(obj runtime.Object, to unversioned.GroupVersion) (url.Values, error) { ret, err := api.ParameterCodec.EncodeParameters(obj, to) if err != nil && runtime.IsNotRegisteredError(err) { // fallback to v1 return api.ParameterCodec.EncodeParameters(obj, v1.SchemeGroupVersion) } return ret, err } func (versionedParameterEncoderWithV1Fallback) DecodeParameters(parameters url.Values, from unversioned.GroupVersion, into runtime.Object) error { return errors.New("DecodeParameters not implemented on versionedParameterEncoderWithV1Fallback") } // VersionedParameterEncoderWithV1Fallback is useful for encoding query // parameters for thirdparty resources. It tries to convert object to the // specified version before converting it to query parameters, and falls back to // converting to v1 if the object is not registered in the specified version. // For the record, currently API server always treats query parameters sent to a // thirdparty resource endpoint as v1. var VersionedParameterEncoderWithV1Fallback runtime.ParameterCodec = versionedParameterEncoderWithV1Fallback{} </ins>

It seems to me like all of the garbage collection integration tests produce a ton of log messages of the form:


Question: local output=`check-minion ${minion_ip}` echo $output if [[ "${output}" != "working" ]]; then <del> if (( attempt > 9 )); then </del> <ins> if (( attempt > 20 )); then </ins> echo echo -e "${color_red}Your cluster is unlikely to work correctly." >&2 echo "Please run ./cluster/kube-down.sh and re-create the" >&2

When creating a new cluster on AWS with the 1.2 version I always get a failure when checking docker on the nodes, but after the check loop ends everything seems to work fine and all nodes are marked as ready. The instance types for nodes is
That is very surprising. Are you using any particular settings (to help me try to reproduce the problem)?
These are the exports I make before running kube-up:
The problem is the instances take a very long time to start up. I had to increase the timeout to 19.
Hey how did you increase the timeout? Didn't see a setting in config- that I could override. Getting the same issue as the poster.
Thanks , that did the trick!


Question: }) ginkgo.BeforeEach(func() { <del> if err := lookEmitSignalCommand(); err != nil { e2eskipper.Skipf("skipping test because: %v", err) return } </del> ginkgo.By("Wait for the node to be ready") waitForNodeReady()

xref: Graceful node shutdown test uses to invoke dbus - As per some distros, don't have installed so the test must be skipped in those environments. We should consider if we can switch to use or perhaps and fall back to to ensure the test can work across more distros.
/sig node
/sig testing
/remove-kind bug /kind feature I'm putting kind feature to signify that nothing is really broken in k8s, tests needs improvement.
/remove-sig testing This dbus command should be up to SIG node, though tentatively: /triage accepted
/area test
Do we actually have these other distros available to prevent regressions if we change that?
/assign
the change will be to replace skip logic when is missing to use an alternative way to send this. So there will be no regression on distros where it currently runs without the skip. Original bug should have a link to job that was failing
I would like to work on improving the command. Am I understanding correct here We have to change to logic where we skip to use busctl instead. Could you please give me some idea on how can we make it to use , some similar examples in the repo would be really useful.
please have a look at The issue is that the command available is too old and doesn't include .
I see, it looks like we are not just adding a condition for instead we are replacing it with . It looks you are already working on it. Please inform if I can help in anyway. Thanks for the response.
Both are possible... it was easier to get a CI signal if that was used in all cases. I might put the condition later. I am waiting for comments in the PR, but feel free to propose other solutions that might work with older systemd versions.


Question: } patchBytes, patchObject, err = p.patchSimple(current, modified, source, namespace, name, errOut) } <del> if err != nil && errors.IsConflict(err) && p.force { </del> <ins> if err != nil && (errors.IsConflict(err) || errors.IsInvalid(err)) && p.force { </ins> patchBytes, patchObject, err = p.deleteAndCreate(current, modified, namespace, name) } return patchBytes, patchObject, err

/kind bug What happened: From I expected kubectl apply to support deleting a resource and recreating it, if changes cannot be applied. kubectl replace works: What you expected to happen: How to reproduce it (as minimally and precisely as possible): ` Change clusterIP to any IP address. Anything else we need to know?: Environment: Kubernetes version (use ): client v1.11.0, cluster v1.11.1 Cloud provider or hardware configuration: QEMU/KVM OS (e.g. from /etc/os-release): Cluster CentOS Linux release 7.5.1804 (Core), client macOS Kernel (e.g. ): 3.10.0-862.6.3.el7.x86_64 Install tools: kubeadm Others:
/cc
/assign


Question: Filter: v1beta2.PluginSet{ Enabled: []v1beta2.Plugin{ {Name: "Plugin1"}, <ins> {Name: "Plugin2"}, {Name: "Plugin3"}, </ins> }, }, },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Recent commit: import an issue when multiple custom plugin configured in scheduler config file, and those plugins are default plugins too. It was manipulating slice with index, but the slice is updated while index is not, which will cause the index out of range, see the exceptions listed below, config file should be handled properly without such exception override multiple default plugin in scheduler's config file. Kubernetes version (use ): latest (1.22) Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign
/sig scheduling /triage accepted


Question: print_codegens exit 1 fi <del> codegens_to_run+=("${matches[@]}") </del> <ins> # The array-syntax abomination is to accommodate older bash. codegens_to_run+=("${matches[@]:+"${matches[@]}"}") </ins> done # If no codegens were specified, run them all.

I am working on a PR where I am removing some legacy code from a API. I believe I have to run some scripts to generate code. When I run I get the following error: I expect the script to work and run without error. Run hack/update- or run I am on the latest build. <details</details<details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc are you synced to HEAD of master? what version of bash are you running?
I am synced to HEAD of master and my bash version is version 4.2.46(2).
Pretty sure identified it correctly as a bash version issue. I ran into this issue and it was due to using an older bash. I know that 4.2.46 (which is used in AL2) is too old, but I'm not sure exactly what the minimum version is. If you are using AL2 for development, this will drop in a compatible version:
/kind support
Let me see if there's a way to make old-bash shut up. Sadly I do not have an old version to try against - - are you willing to type commands into your shell for me?
Yep, you just caught me before I purged my old environment with suggestions.
would a slack conversation be easier?
maybe but I am juggling meetings :)
There are a few problems I ran into before just deciding to upgrade to bash v5, but I think this initial thing was a bug fixed in bash 4.4+: From, when the nounset option is enabled no longer throws an unbound variable error. So on 4.2, this fails:
working on a patch


Question: } // TODO: if we ever want to go generic, this allows a clean -o yaml without trying to print columns or anything // rawExtension := &runtime.Unknown{ <del> // 	Raw: originalPatchedObjJS, </del> <ins> //	Raw: originalPatchedObjJS, </ins> // } printer, err := f.PrinterForMapping(cmd, mapping, false)

Hi, Deployment patching with doesn't show up in the deployment rollout history: first: after: Is it a bug? Should it show up? Thanks, Peter
cc
Which kubernetes and kubectl releases? Did the patch actually trigger a rollout?
I also run the command from CI with the latest . So the problem affects the v1.2.3 kubectl as well
We should also fix the way we sort revisions - 10 should be after 9, not 1.
Can confirm this. The issue is that the patch isn't updating , so the rollout history will just copy the value from the previous version. If you started with you get that for the patches. In my case I have a different line repeated. Builds 7 and 8 were built with different . I'm not sure why the patch worked once. Command I am calling is:
If no one is working on patch - i can try to make one.
I'll like to work on this issue if you haven't started yet
Actually never mind I can look at something else :)
Hi, i see that you are the author of the following comment. I think that currently every valid change cause is discarded because of version conflict. Maybe you have some information why it happens?
command instead generates a patch to update then applies that.
It was the same for patch, but for some reason was changed.
The intent was two fold. First, prevent the change-cause from ever failing patch the CLI command. change-cause isn't authoritative and the patch took hold, so a non-zero return code is inappropriate. Second, if the resource version change post-patch, before the update, the change-cause is unknown, something else touched it after the fact. A command that issues a instead of an could very easily break the history since patches will attempt to reapply themselves regardless of resource version in the face of conflicts. The object returned from the patch operation is the one that we use to avoid hitting a resource conflict with ourselves.

Truth be told, I don't feel very strongly about getting the history right since it can be changed anyone with write access to the object, includes secrets that should be elided (--token for instance), ignores environment variables which affect outcome, ignores the actual path used to locate the binary you're running, and references paths that are only good from one location on one machine, so if you want to switch it back to a second patch, I don't mind as long as a failure to update the object doesn't cause a non-zero return from the command.
Could there be a race between the call and the deployment updating itself to action the changes? On 19 May 2016 9:47 pm, "David Eads" wrote:


Question: }, }), }, <ins> "bad-volume-zero-capacity": { isExpectedFailure: true, volume: testVolume("foo", "", core.PersistentVolumeSpec{ Capacity: core.ResourceList{ core.ResourceName(core.ResourceStorage): resource.MustParse("0"), }, AccessModes: []core.PersistentVolumeAccessMode{core.ReadWriteOnce}, PersistentVolumeSource: core.PersistentVolumeSource{ HostPath: &core.HostPathVolumeSource{ Path: "/foo", Type: newHostPathType(string(core.HostPathDirectory)), }, }, }), }, </ins> "missing-accessmodes": { isExpectedFailure: true, volume: testVolume("goodname", "missing-accessmodes", core.PersistentVolumeSpec{

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT /kind bug What happened: If someone requests a zero (0) capacity PVC and an administrator creates a zero (0) capacity PV, Kubernetes will happily bind them together. Unfortunately, any use of this PVC/PV combination by a pod will fail. It fails because the zero capacity PV cannot be formatted or mounted successfully. Note: The same issue happens for dynamically provisioned PVs. What you expected to happen: I expect that Kubernetes would not allow creation of invalid PVC and PV objects, aka zero (0) capacity PVs and PVCs. How to reproduce it (as minimally and precisely as possible): Create a pod that requests a zero capacity PVC, using dynamic PV provisioning is simpler than creating your own zero (0) capacity PV. The pod will fail to reach a running state, since the related PV fails to be formatted and mounted successfully. Here is the error that I received for a test I performed requesting a zero capacity PVC using ceph rbd: Anything else we need to know?: Here is a related item with a specific fix for external ceph rbd provisioner: Environment: Nothing specific to environment
/sig storage
/assign


Question: "k8s.io/kubernetes/pkg/volume/util/types" ) <ins> func pluginPVOmittingClient(dswp *desiredStateOfWorldPopulator) { fakeClient := &fake.Clientset{} fakeClient.AddReactor("get", "persistentvolumeclaims", func(action core.Action) (bool, runtime.Object, error) { return false, nil, nil }) fakeClient.AddReactor("get", "persistentvolumes", func(action core.Action) (bool, runtime.Object, error) { return false, nil, nil }) dswp.kubeClient = fakeClient } func prepareDswpWithVolume(t *testing.T) (*desiredStateOfWorldPopulator, kubepod.Manager) { // create dswp mode := v1.PersistentVolumeFilesystem pv := &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: "dswp-test-volume-name", }, Spec: v1.PersistentVolumeSpec{ ClaimRef:   &v1.ObjectReference{Namespace: "ns", Name: "file-bound"}, VolumeMode: &mode, }, } pvc := &v1.PersistentVolumeClaim{ Spec: v1.PersistentVolumeClaimSpec{ VolumeName: "dswp-test-volume-name", }, Status: v1.PersistentVolumeClaimStatus{ Phase: v1.ClaimBound, }, } dswp, fakePodManager, _ := createDswpWithVolume(t, pv, pvc) return dswp, fakePodManager } func TestFindAndAddNewPods_WithReprocessPodAndVolumeRetrievalError(t *testing.T) { // create dswp dswp, fakePodManager := prepareDswpWithVolume(t) // create pod containers := []v1.Container{ { VolumeMounts: []v1.VolumeMount{ { Name:      "dswp-test-volume-name", MountPath: "/mnt", }, }, }, } pod := createPodWithVolume("dswp-test-pod", "dswp-test-volume-name", "file-bound", containers) fakePodManager.AddPod(pod) podName := util.GetUniquePodName(pod) dswp.findAndAddNewPods() if !dswp.podPreviouslyProcessed(podName) { t.Fatalf("Failed to record that the volumes for the specified pod: %s have been processed by the populator", podName) } pluginPVOmittingClient(dswp) dswp.ReprocessPod(podName) dswp.findAndAddNewPods() if !dswp.podPreviouslyProcessed(podName) { t.Fatalf("Failed to record that the volumes for the specified pod: %s have been processed by the populator", podName) } fakePodManager.DeletePod(pod) } func TestFindAndAddNewPods_WithVolumeRetrievalError(t *testing.T) { // create dswp dswp, fakePodManager := prepareDswpWithVolume(t) pluginPVOmittingClient(dswp) // create pod containers := []v1.Container{ { VolumeMounts: []v1.VolumeMount{ { Name:      "dswp-test-volume-name", MountPath: "/mnt", }, }, }, } pod := createPodWithVolume("dswp-test-pod", "dswp-test-volume-name", "file-bound", containers) fakePodManager.AddPod(pod) podName := util.GetUniquePodName(pod) dswp.findAndAddNewPods() if dswp.podPreviouslyProcessed(podName) { t.Fatalf("The volumes for the specified pod: %s should not have been processed by the populator", podName) } if dswp.podHasBeenSeenOnce(podName) { t.Fatalf("The volumes for the specified pod: %s should not have been processed by the populator", podName) } } </ins> func TestFindAndAddNewPods_FindAndRemoveDeletedPods(t *testing.T) { // create dswp mode := v1.PersistentVolumeFilesystem

It happens during the "delete" phase in the load test. ! In perf-dash we can see that in total there are ~20M requests like that during the load test. ! It started in Sep 2019 (sic!), so probably when we enabled PVs in our load tests. We haven't detected it so far as, strangely, it doesn't seem to be impacting our scalability SLIs. We should investigate this. /sig scalability
I checked and we create 35 PersistentVolumeClaims per 100 nodes. So in case of 5K node test there will be 1750 PVCs in the cluster. This is close to the 2K of GET PVC requests we see.
Let's analyze the last run of 5K node test in more depths - Grafana: ! As I wrote already somewhere here, it happens during the delete phase. We reach the peak of ~2K QPS around 2:30 AM UTC in that run. Let's check the logs (api-server, kubelets) around that time.
Looking at the api-server logs one can easily tell that something is wrong. It's swamped by GET persistentvolumeclaim requests:
Restricting the logs to a single stateful set on a single node (single client ip) - it also doesn't look right. Total such requests from this IP address: So it looks like a single kubelet issued almost 20K GET requests for this PVC within ~75min, which gives us ~4 QPS from a single kubelet...
From the scheduler logs it looks like the statefulset pod for this PVC was bound on the node Let's check the kubelet logs there
Kubelet logs: Bingo, around the time when the issue happens, it's full of the following logs:
Looks like the errors disappeared exactly when the pod was deleted So, what happens here is partially caused by our test as it may happen that we'll first delete PVC and won't delete the corresponding pod for the next O(1h). Nevertheless, the code in the should be fixed so it doesn't cause kubelet to send ~4QPS of GET PVC requests in such case.
/sig node /sig storage
There is a pending discussion about ways to fix it in
Thanks for fixing that! I know how much work was it and I really appreciate!


Question: {testName: "supported version", ver: &csipb.Version{Major: 0, Minor: 0, Patch: 0}}, {testName: "supported version", ver: &csipb.Version{Major: 0, Minor: 1, Patch: 0}}, {testName: "supported version", ver: &csipb.Version{Major: 0, Minor: 1, Patch: 10}}, <ins> {testName: "supported version", ver: &csipb.Version{Major: 0, Minor: 2, Patch: 0}}, </ins> {testName: "supported version", ver: &csipb.Version{Major: 1, Minor: 1, Patch: 0}}, {testName: "supported version", ver: &csipb.Version{Major: 1, Minor: 0, Patch: 10}}, {testName: "unsupported version", ver: &csipb.Version{Major: 10, Minor: 0, Patch: 0}, mustFail: true},

Current CSI E2E test has some issues which prevent it to run successfully the a local cluster. /sig storage /kind bug /assign


Question: apiextensions.SetCRDCondition(crd, establishedCondition) updatedObj, err := c.crdClient.CustomResourceDefinitions().UpdateStatus(crd) <ins> if apierrors.IsNotFound(err) || apierrors.IsConflict(err) { // deleted or changed in the meantime, we'll get called again return nil } </ins> if err != nil { return err }

k8s version: v1.9.3 When I create customresourcedefinition, There is error like Operation cannot be fulfilled on "": the object has been modified; please apply your changes to the latest version and try again . apiVersion: kind: CustomResourceDefinition metadata: name: annotations: "": crd-install labels: app: istio-pilot spec: group: names: kind: VirtualService listKind: VirtualServiceList plural: virtualservices singular: virtualservice categories: istio-io networking-istio-io scope: Namespaced version: v1alpha3 So what is the reason of it?
a few questions: where are you seeing the error, in the apiserver log or in the output of the kubectl command? what kubectl command are you using to create the object? are you running HA apiservers?
The log is shown by apiserver. I run k8s in HA apiservers. The command I used is "kubectl apply -f "+file.
API servers attempt to update CRD status to indicate whether the names were available. In an HA apiserver setup, the API servers race to do so, and all but the first lose the race with a resource conflict error. That is not unexpected. Opened to avoid logging normal API responses in that scenario.
Thanks ! But I want to know whether the CRD has been set successfully in HA apiservers ? I use the command " kubectl get crd -o yaml" and get the following results. ! I see its status is true. But when i create a virtualservices, it returns error. Error from server (Timeout): error when creating "cms-api-": Timeout: request did not complete within allowed duration ! So i want to know how to solve this error.
/assign
Excuse me. Do you have some greate ideas to solve this issue??
There is a PR for this sent by
I see your request is timed-out not failed. In HA, I expect the request to fail in the first seconds of the CRD creation, but after that it should go through. I need more logs during this timeout or a consistent way to reproduce it.


Question: The `KUBERNETES_PROVIDER` environment variable tells all of the various cluster management scripts which variant to use.  If you forget to set this, the assumption is you are running on Google Compute Engine. <ins> By default, the Vagrant setup will create a single master VM (called kubernetes-master) and one node (called kubernetes-node-1). Each VM will take 1 GB, so make sure you have at least 2GB to 4GB of free memory (plus appropriate free disk space). Vagrant will provision each machine in the cluster with all the necessary components to run Kubernetes.  The initial setup can take a few minutes to complete on each machine. </ins> If you installed more than one Vagrant provider, Kubernetes will usually pick the appropriate one. However, you can override which one Kubernetes will use by setting the [`VAGRANT_DEFAULT_PROVIDER`](https://docs.vagrantup.com/v2/providers/default.html) environment variable: ```sh

I cannot find such a service when . Same thing applies for . My cluster runs fine. cc:
Ah, good catch... everything is run as a container now, will update the documentation on where to get the logs.


Question: kubectl delete clusterrolebinding kube-aggregator:system:kube-aggregator > /dev/null 2>&1 || true kubectl create clusterrolebinding kube-aggregator:system:auth-delegator --clusterrole=system:auth-delegator --serviceaccount=kube-public:kube-aggregator kubectl create clusterrolebinding kube-aggregator:system:kube-aggregator --clusterrole=system:kube-aggregator --serviceaccount=kube-public:kube-aggregator <del> kubectl delete rolebinding kube-aggregator:authentication-reader > /dev/null 2>&1 || true </del> <ins> kubectl delete rolebinding -n kube-system kube-aggregator:authentication-reader > /dev/null 2>&1 || true </ins> kubectl create rolebinding -n kube-system kube-aggregator:authentication-reader --role=extension-apiserver-authentication-reader --serviceaccount=kube-public:kube-aggregator # make sure the resources we're about to create don't exist

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG Create/delete rolebinding kube-aggregator:authentication-reader in different namespace cause to the failure detail:


Question: elapsedSeconds := elapsed.Seconds() cleanContentType := cleanContentType(contentType) requestCounter.WithLabelValues(reportedVerb, dryRun, group, version, resource, subresource, scope, component, cleanContentType, codeToString(httpCode)).Inc() <ins> // MonitorRequest happens after authentication, so we can trust the username given by the request info, ok := request.UserFrom(req.Context()) if ok && info.GetName() == user.APIServerUser { apiSelfRequestCounter.WithLabelValues(reportedVerb, resource, subresource).Inc() } </ins> if deprecated { deprecatedRequestGauge.WithLabelValues(group, version, resource, subresource, removedRelease).Set(1) audit.AddAuditAnnotation(req.Context(), deprecatedAnnotationKey, "true")

Currently we don't have an easy way of determining what number of requests are apiserver self-requests. This has historically bitten us, since we used to have a 50(?) QPS limit on apiserver self-requests (which thus has been removed). Even so, it would be a good idea to know this generally, since this may be helpful in pinpointing performance regressions for apiserver. /sig api-machinery instrumentation
Can this issue be considered a newbie one? I'm hunting for issues which might be a good first PR.
In the triage meeting was mentioned that would be good to have similar issues to track other well known actors like controller-manager and kubelet. But let's keep this one for api-server calling itself to have it done. /cc
/cc


Question: } } <del> return c.updateInstanceSecurityGroupsForNLB(loadBalancerName, nil, nil, nil) </del> <ins> return c.updateInstanceSecurityGroupsForNLB(loadBalancerName, nil, nil, nil, nil) </ins> } lb, err := c.describeLoadBalancer(loadBalancerName)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For each NLB created, K8s creates an ingress rule on the node group's security group for each port exposed for each CIDR attached to the VPC to allow health checks to succeed. AWS limits the number of rules per security group to 60 by default. This quota is adjustable but multiplied by the quota for security groups per network interface cannot exceed 1000. I have a VPC with 16 CIDR blocks attached to it and a quota for rules per security group set to 80. Each NLB created adds 18 new rules (16 for Health Check, 1 for Client, and 1 for MTU) to the node group's security groups. This high number of unnecessary rules created makes it impossible to create more than 4 NLBs per node group (if each one has a single port exposed). What you expected to happen: For the health check to succeed, the NLB must be able to ping the node port for the service on the cluster's node group. So the node group's security group just needs an inbound rule for each ENI of the NLB, making the number of rules necessary on the cluster's node group directly proportional to the number of AZs the NLB is configured. How to reproduce it (as minimally and precisely as possible): Create a VPC on AWS with 8 CIDRs attached to it; Create a K8s Cluster; Create a Network Load Balancer with 6 ports exposed; If the default quota of rules per security group has not been changed, the NLB creation will fail with RulesPerSecurityGroupLimitExceeded. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/area provider/aws