Question: if len(v) > 0 { out, err := b.exec.Run("iscsiadm", "-m", "discoverydb", "-t", "sendtargets", "-p", tp, "-I", b.Iface, "-o", "update", "-n", k, "-v", v) if err != nil { <del> return fmt.Errorf("iscsi: failed to update discoverydb key %q with value %q error: %v", k, v, string(out)) </del> <ins> return fmt.Errorf("iscsi: failed to update discoverydb key %q error: %v", k, string(out)) </ins> } } }

This issue was reported in the Description Kubernetes can be configured to use iSCSI volumes. When using CHAP authentication, CHAP secrets are stored using the Secrets API, such as in this . When a pod is configured to use iSCSI and the AttachDisk method is called, this will call the code in Figure 1. Figure 19.1: iSCSI secret handling These two functions both iterate over a slice of strings that are keys that reference secrets in a map. These are then used to generate iscsiadm commands. As shown, if there are errors in executing these commands, errors are returned with both the key and secret values in the error string. These errors will eventually be logged using klog: Someone with access to these logs would be able to view the sensitive secrets and could potentially gain access to iSCSI volumes. Exploit Scenario Alice runs a cluster, and wishes to use iSCSI for data storage. Eve has access sufficient to collect the logs, and uses this access to connect to iSCSI storage devices as a privileged user. Recommendation Short term, as in TOB-K8S-001: Bearer tokens revealed in logs, do not log sensitive credentials at any logging level, as they may accidentally leak into inappropriate environments, such as production. Long term, implement policies that enforce code review to ensure that sensitive data is not exposed in logs, or implement logging filters that check for sensitive data and remove it prior to reification within logs. In either case, ensure that sensitive data cannot be stored in logs. Anything else we need to know?: See for current status of all issues created from these findings. The vendor gave this issue an ID of ATR-K8S-003 and it was finding 22 of the report. The vendor considers this issue Medium Severity. To view the original finding, begin on page 61 of the Environment: Kubernetes version: 1.13.4
/sig storage /area security /kind bug /remove-kind feature


Question: # Make /tmp hermetic build --sandbox_tmpfs_path=/tmp <ins> # This tag is required to build github.com/opencontainers/selinux correctly build --define gotags=selinux </ins> # Ensure that Bazel never runs as root, which can cause unit tests to fail. # This flag requires Bazel 0.5.0+ build --sandbox_fake_username

What happened: After upgrading from kubelet 1.15 -1.16, volumes are no longer relabeled as containerfilet for selinux, causing most pods to fail. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Run a kubernetes cluster on 1.15.[0-3]. Create a pod with a secret, configmap, or with provisioned storage (ebs-csi-driver). Find that file on disk of the node, and observe it is labeled with the file type containerfilet and the pod is able to read it. Upgrade the kubelet to 1.16 Delete the pod, and observe it fail on launch. Find the file on disk and see it labeled tmpfst or unlabeledt for configmap/secret and provisioned volumes, respectively. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 / 1.16 Cloud provider or hardware configuration: AWS OS (e.g: ): RHEL 7.7 Kernel (e.g. ): 3.10.0-1062.1.1.el7.x86_64 Install tools: kubeadm Network plugin and version (if this is a network-related bug): N/A Others:
/sig storage
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Is selinux enabled for docker? Can you paster output of : also
Also please paste YAML of one or more pods which are affected by this. Thank you!
Security Options - Perhaps of note here, we do not change the installed by docker, and confirms we don't make changes to that file. Docker version: Client and server blocks match. Pod yaml incoming
To clarify, do you want a runnable "minimum reproducible" , or is it more informational? If the latter, I have a pod that breaks and I'm working on the yaml for that, removing sensitive information, etc. If the former, I'll need to spend a few minutes getting a minimal reproduction with busybox or somesuch.
I need full pod's yaml that is not working. i.e - generated full YAML.

Hi, I'm sorry I haven't had a chance to follow up with this ticket. I will get some pod yaml attached early next week. Thank you for following up.
I think I'm seeing the same thing, to reproduce, use the following resource YAML: On k8s v1.15.5 (hyperkube) On k8s 1.16.2 (hyperkube) Are you in Enforcing mode () ?
Hey thanks much for following on with example YAML. We've been totally swamped with work so I haven't had time to address this issue. To answer your question: we were seeing all pods with storage (configmaps, secrets, persistent volumes) failing in Enforcing mode, and disabling Enforcement () allowed the same pods to successfully run, with no problem. There was not as much output in audit logs as we would have expected, so constructing custom policies to allow access was a bit of shooting in the dark. We were able to get configmaps and secrets to work passably with a custom policy allowing limited access to tmpfs labeled items, but we can't get away with allowing access to unlabeledt in our compliance environment. Thanks again for following up with more info and linking up this ticket, and let me know if I can provide any more helpful information. We've rolled back to 15.3 for the time being, so I'll be constructing knowledge somewhat from memory, but the upgrade was run recently, and I took reasonably detailed notes, so my info is be largely accurate.
cc
Can someone post pod's YAML after pod was created. I am looking for some generated fields in the Pod.
I finally had the chance today to dig a bit deeper why SELinux is broken in 1.16+ In the linked PR you can see the fix. See the details in there. If you want to give that a try, I have build a binaries here: A workaround I have used in the meantime was: My reproduction of the bug can be found there: If you run the packer build the pods fail because of the SELinux problem
nice work! looks like the patch we need is this one right?
if so, can you please propose a PR against master of kubernetes/kubernetes?
Reopening to track adding a test.
On second thought, spawned a separate issue to track adding test coverage. /close
Closing this issue.

<detailsIn response to : On second thought, spawned a separate issue to track adding test coverage. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
we have just updated our cluster to 1.16.5, and even 1.16.6-beta.0, but the secret volume is still labeled as tmpfst:
I see the same result on the test above on v1.17.2
I can confirm the release binaries are not build correctly I have opened this issue in the release repo: /reopen Workaround is building the binaries yourself using bazel and the release tag
Reopened this issue. <detailsIn response to : I can confirm the release binaries are not build correctly I have opened this issue in the release repo: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can someone confirm that 1.16.7 / 1.17.3 binaries are built with the selinux tags in make targets?
No they haven't as haven't been merged. Feel free to ask on those PRs again, I have not been successful, convincing that this should be merged with at least some priority
I am able to run this repro and get the desired outcome on 1.17.4. However, I am trying to attach Cinder volume whose context doesn't automatically get relabeled as containerfilet when I attempt to attach volumes with both in-tree and Cinder CSI plugin. I have filed this bug in the cloud-provider-openstack repo but not sure if this is a Kubernetes problem or a cinder-csi-plugin problem:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /assign are looking at adding selinux to CI using kops
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: serv.Timeout = uint32(svcInfo.StickyMaxAgeSeconds) } if err := proxier.syncService(svcNameString, serv, true); err == nil { <ins> // check if service need skip endpoints that not in same host as kube-proxy onlyLocal := svcInfo.SessionAffinityType == api.ServiceAffinityClientIP && svcInfo.OnlyNodeLocalEndpoints </ins> activeIPVSServices[serv.String()] = true <del> if err := proxier.syncEndpoint(svcName, svcInfo.OnlyNodeLocalEndpoints, serv); err != nil { </del> <ins> if err := proxier.syncEndpoint(svcName, onlyLocal, serv); err != nil { </ins> glog.Errorf("Failed to sync endpoint for service: %v, err: %v", serv, err) } } else {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The test is flaky with the error something like: What you expected to happen: Test should pass /assign
/sig network /area ipvs
/close
Cases are green now. :tada:


Question: <ins> apiVersion: v1 kind: Pod metadata: name: dns-frontend labels: name: dns-frontend spec: containers: - name: dns-frontend image: k8s.gcr.io/example-dns-frontend:v1 command: - python - client.py - http://dns-backend.development.svc.cluster.local:8000 imagePullPolicy: Always restartPolicy: Never </ins>

What happened: We've been doing some release readiness and auditing all tagged ginkgo tests... Looks like The test seems to expect a GOPATH when running. Im seeing some failures in sonobuoy and suspect a possible relation there but havent dove into it. Either way, its clear this test might need some updating/fixing/cleanuping and parts can be rewritten/improved. What you expected to happen: No reliance on GOPATH for an e2e test ... I dont remember the whole history on this, but no tests that rely on static files currently look in gopath, from what i can tell ? instead we plumbed in a repo-root/ option a few years ago. There may be a state of the art in terms of how we do this.... but i thought it was at one point which was an option passed in. How to reproduce it (as minimally and precisely as possible): Try to run this in sonobuoy or any other environment where there's no GOPATH or non-standard source trees, pretty sure it might not work. Anything else we need to know?: I think the fix here is to just embed the YAML in golang like we do in other tests. along the way there's other cleanup we can do to improve this test. Environment: Kubernetes version (use ): master
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
the YAML files are pretty small, so +1 to embed them in i don't think testing the YAML from the separate repository () directly is required here. ref /help /kind cleanup /priority backlog
/remove-help
/assign Also I should say, I talked to Jay on slack and he is ok with me taking a look at it, if thats ok.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle stale
/remove-lifecycle stale
xref: go 1.16 has built in support for embedding go 1.17 looks to be removing GOPATH
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/help wanted
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten still a problem. FYI re: GOPATH removal
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/assign


Question: } // Exists checks if a certificate exist <del> func (rw *pkiCertificateReadWriter) Exists() bool { </del> <ins> func (rw *pkiCertificateReadWriter) Exists() (bool, error) { </ins> certificatePath, _ := pkiutil.PathsForCertAndKey(rw.certificateDir, rw.baseName) return fileExists(certificatePath) } <del> func fileExists(filename string) bool { </del> <ins> func fileExists(filename string) (bool, error) { </ins> info, err := os.Stat(filename) <del> if os.IsNotExist(err) { return false </del> <ins> if err != nil { if os.IsNotExist(err) { return false, nil } return false, err </ins> } <del> return !info.IsDir() </del> <ins> return !info.IsDir(), nil </ins> } // Read a certificate from a file the K8s pki managed by kubeadm

When the command is run with superuser privileges, it prints the requested information. Print a warning that files could not be read due to insufficient permission, or that information was missing (e.g., when the /etc/kubernetes directory is not initialized). When the directory is readable, it works as expected: where OS and kubeadm version detail was: Issue the command as a normal user when the directory /etc/kubernetes/pki exists but not readable. No response <details</details<detailsBare metal </details<details</details<detailsapt-get install kubeadm (source repo: ) </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle


Question: } // Setup event recorder if required. <del> makeEventRecorder(&s.KubeletConfiguration, kubeDeps, nodeName) </del> <ins> makeEventRecorder(kubeDeps, nodeName) </ins> if kubeDeps.ContainerManager == nil { if s.CgroupsPerQOS && s.CgroupRoot == "" {

/approve no-issue isn't working on , so I'm trying this instead. /cc
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
thanks


Question: timeout := 10 * time.Minute startTime := time.Now() <del> allPods := getAllPods(c) scheduledSystemPods, currentlyNotScheduledSystemPods := getSystemPods(c) systemPods := scheduledSystemPods + currentlyNotScheduledSystemPods // Wait for system pods to be scheduled, and for pods in all other namespaces to be deleted for currentlyNotScheduledSystemPods != 0 || systemPods != allPods { </del> <ins> // Wait for all pods to be scheduled. allScheduledPods, allNotScheduledPods := getFilteredPods(c, masterNodes, metav1.NamespaceAll) for len(allNotScheduledPods) != 0 { </ins> time.Sleep(2 * time.Second) <del> scheduledSystemPods, currentlyNotScheduledSystemPods := getSystemPods(c) systemPods = scheduledSystemPods + currentlyNotScheduledSystemPods allPods = getAllPods(c) </del> if startTime.Add(timeout).Before(time.Now()) { <ins> framework.Logf("Timed out waiting for the following pods to schedule") for _, p := range allNotScheduledPods { framework.Logf("%v/%v", p.Namespace, p.Name) } </ins> framework.Failf("Timed out after %v waiting for stable cluster.", timeout) break } <ins> allScheduledPods, allNotScheduledPods = getFilteredPods(c, masterNodes, metav1.NamespaceAll) </ins> } <del> return scheduledSystemPods </del> <ins> return len(allScheduledPods) </ins> } <del> // getAllPods lists all pods in the cluster, with succeeded and failed pods filtered out and returns the count func getAllPods(c clientset.Interface) int { allPods, err := c.CoreV1().Pods(metav1.NamespaceAll).List(metav1.ListOptions{}) </del> <ins> // getFilteredPods lists scheduled and not scheduled pods in the given namespace, with succeeded and failed pods filtered out. func getFilteredPods(c clientset.Interface, masterNodes sets.String, ns string) (scheduledPods, notScheduledPods []v1.Pod) { pods, err := c.CoreV1().Pods(ns).List(metav1.ListOptions{}) </ins> framework.ExpectNoError(err, "listing all pods in kube-system namespace while waiting for stable cluster") // API server returns also Pods that succeeded. We need to filter them out. <del> currentPods := make([]v1.Pod, 0, len(allPods.Items)) for _, pod := range allPods.Items { </del> <ins> filteredPods := make([]v1.Pod, 0, len(pods.Items)) for _, pod := range pods.Items { </ins> if pod.Status.Phase != v1.PodSucceeded && pod.Status.Phase != v1.PodFailed { <del> currentPods = append(currentPods, pod) </del> <ins> filteredPods = append(filteredPods, pod) </ins> } } <del> allPods.Items = currentPods return len(allPods.Items) } // getSystemPods lists the pods in the kube-system namespace and returns the number of scheduled and unscheduled pods func getSystemPods(c clientset.Interface) (int, int) { systemPods, err := c.CoreV1().Pods(metav1.NamespaceSystem).List(metav1.ListOptions{}) framework.ExpectNoError(err, "listing all pods in kube-system namespace while waiting for stable cluster") scheduledSystemPods, currentlyNotScheduledSystemPods := e2epod.GetPodsScheduled(masterNodes, systemPods) return len(scheduledSystemPods), len(currentlyNotScheduledSystemPods) </del> <ins> pods.Items = filteredPods return e2epod.GetPodsScheduled(masterNodes, pods) </ins> }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Conformance-GCE-master Which test(s) are failing: Since when has it been failing: 11-07 14:41 PST probably after this PR got merged Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /cc /sig scheduling /milestone v1.17 /priority critical-urgent
/assign draveness ahg-g
it is actually after , which was trying to fix WaitForStableCluster is timing out, sadly there are not enough logs from the test to understand why, but likely because there are pods in a namespace other than kube-system from previous tests that never get deleted.
oh I see... so you think it's a flake? Or we need one more fix to the WaitForStableCluster?
We need to fix WaitForStableCluster again. /cc
Thanks for such a quick turnover!


Question: limits := core.ResourceList{} zeroQuantity := resource.MustParse("0") isGuaranteed := true <del> for _, container := range pod.Spec.Containers { </del> <ins> allContainers := []core.Container{} allContainers = append(allContainers, pod.Spec.Containers...) allContainers = append(allContainers, pod.Spec.InitContainers...) for _, container := range allContainers { </ins> // process requests for name, quantity := range container.Resources.Requests { if !isSupportedQoSComputeResource(name) {

What happened: A pod that has init container with request for cpu and memory, and containers that make no request for cpu or memory reports a BestEffort QoS class. This is incorrect. What you expected to happen: Containers that make requests independent of lifecycle phase should impact quality of service calculation. Found in 1.12, looks to still be present in 1.14 dev. /cc
/priority important-soon
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
I think this doesn't work very well with the current cpu manager logic. The problem is that a pod can't now have QoS class Guaranteed if it has init containers which don't have matching requests/limits. This means that normal containers in such a pod don't then get dedicated cpus even though the init containers are long gone when the app containers (with matching requests/limits) are finally started. This is quite non-intuitive. I would propose separating at least Guaranteed class calculation so that the init container resource requests wouldn't affect the calculation. What do you think?


Question: e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2eruntimeclass "k8s.io/kubernetes/test/e2e/framework/node/runtimeclass" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" <ins> e2epv "k8s.io/kubernetes/test/e2e/framework/pv" </ins> e2erc "k8s.io/kubernetes/test/e2e/framework/rc" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" testutils "k8s.io/kubernetes/test/utils"

Since v1.27.0, the for appear to have extra validation on whether the values exist, despite using the operator. This works as expected on 1.26.14, and no longer works on 1.27.0 onwards (tested up to 1.29.2). I am unsure whether this is an intentional breaking change in 1.27.0 or something wrong in our manifests and we've just so happened to get away with it for a few years. The only thing that stands out to me in the changelog for 1.27.0 is some changes to the schedulers plugin but this may well be unrelated. When using the operator, I'd expect the expression to only apply to nodes where the match is truey. Note, using Kind for easy reproduction Since exists in the hostname values of this cluster, the pod schedules as expected. node does not exist in this environment, and so is ignored. Same expectations as 1.26.14, but in this case, the pod stays stuck in pending state due to the following; I have tested this all the way up to 1.29.2. No response v1.27.0 through to v1.29.2 <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
In a similar vein, we have noticed the same issue in K8s v1.27 when the label doesn't match the node name. We see , despite (I think) the nodeinfo util supposed to be called using a node name.
/sig scheduling
This is because the non-existent node is obtained from the cache, which has been fixed by before , get node will return error: after it's fixed since version 1.30, it can run normally PTAL, need to cherry-pick for the previous version?
Looks like my PR accidentally fixed this issue.

is the trigger of this issue, introduced in v1.27; volumebinding plugin just returns node names in PreFilterResult based on nodeAffinity in PV, and thus PreFilterResult could contain node names actually doesn't exist. One Line Summary: If nodeaffinity in PV has non-existing node name in with , the scheduling always fails. And, this bug is already coincidentally fixed by . Questions: In the first place, is it a bug worth cherry-picking? If Yes, should we cherry-pick , which actually isn't the PR motivated for this issue? Or should we go another way
If it's a regression, we should consider cherry-pick. looks big, but the actual code changes are about 10 lines (the rest are comments and tests), so I think it would be ok to cherry-pick. Can you explain why your PR fixes the problem?
Before my PR, we take NodeInfo of nodes in preFilterResult via (). Consequently, we could get not found errors from it when preFilterResult has non-existing node name(s). And, after my PR, we are not using anymore.
The pitfall is that when PreFilterResult was introduced, scheduler directly fetch the nodeNames returned by PreFilter() in its node snapshot, so technically speaking, when a plugin (in-tree or out-of-tree) return non-existent/illegal nodes, the pod scheduling flow will abort immediately. And the PR happened to stomp on this pitfall.
do you mean that you would like to see a smaller version of for cherry-pick? Note that it is only about 10 lines of actual change. The rest is comments and tests. Since this is already released code, I would suggest cherry-picking as-is. Also, worth testing E2E whether the issue reproduces in v1.30. do you think you can give it a try?
yup, i'm in favor of picking the minimum fix, along with a minimum test : b/c in addition to the "fix" part, actually introduced a slight behavior change. do you think the above ^^ minimum fix + makes sense?
can guard it.
Could you let me try it?picking the minimum fix and add e2e test
A simplified version of the sample is used for testing, also get this failedScheduling error, kubectl describe pod pause: !
That small fix should work.

I don't have a strong opinion on whether we should have a minimum fix vs cherry-pick as it is. Defer to you both on that point.
Let's move ahead with the small fix, since we all agree on it.
/reopen Cherry-pick PR(s) aren't done
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
FYI on a related crash
Are there any news on the backporting the solution to older versions (v1.27+)?
This is already backported and released. But the fix for the crash reported in was not released yet. Let's close this issue for now, as it is completed.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The issue here stems from that volume-binding's prefilter assumes that it can equate the matchConstraint hostname labels with node names. This doesn't have to be the case. See here. Prefiltering will always fail in this situation. AFAICT this issue persists with the current head of the master branch.


Question: // initialize ipsetList with all sets we needed proxier.ipsetList = make(map[string]*IPSet) for _, is := range ipsetInfo { <del> if is.isIPv6 { proxier.ipsetList[is.name] = NewIPSet(ipset, is.name, is.setType, isIPv6, is.comment) } proxier.ipsetList[is.name] = NewIPSet(ipset, is.name, is.setType, false, is.comment) </del> <ins> proxier.ipsetList[is.name] = NewIPSet(ipset, is.name, is.setType, isIPv6, is.comment) </ins> } burstSyncs := 2 glog.V(3).Infof("minSyncPeriod: %v, syncPeriod: %v, burstSyncs: %d", minSyncPeriod, syncPeriod, burstSyncs)

/kind bug /area ipv6 /area ipvs /sig network What happened: In an ipv6-only cluster the in mode creates a number of ipset's with . This causes problems and fault logs when they are used with ipv6 addresses. What you expected to happen: The ipset's should be created with . How to reproduce it (as minimally and precisely as possible): Start a ipv6-only cluster Execute; on any node and check the output Anything else we need to know?: The problem seems to be This code seems pretty weird to me. I mean why a flag in the structure at all? I ignored the flag in the structure and just passed the in the directly to and it seem to work. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm on Ubuntu 16.04 OS (e.g. from /etc/os-release): Own BusyBox system Kernel (e.g. ):Install tools: None Others:
Did you make changes like this Earlier error related with ipsets whose is ?
Exactly. I have no idea why any particular ipset would be ipv4 in an ipv6 cluster. But the flag is false for NodePort set's and there is another bug that NodePort addresses are ipv4 (and faulty). So there might have been some special handling of nodeports at some time. IMHO there shoudn't be. FYI, I can't send a PR myself just yet but expect to be able to do that in September.
/cc
I can confirm that, in Ipv6 only cluster, is reporting family inet for many of the entries (KUBE-LOOP-BACK, KUBE-CLUSTER-IP, KUBE-LOAD-BALANCER-FW...).
There is clearly a bug in this code, as it forces IPV4: Should have been an else clause, or just: Even with this, there area entries that are using IPv4. Mostly NodePort related entries. Also see iptables error still:
Correction to above... the ipv6 flag should be based on node's IP family and not the isIPv6 field in the ipsetInfo struct, which has true for all entries, except the nodeport entries. Is anyone planning on creating a PR for this?

Looking forward to PRs
The following codes actually act the same because are of type , which have only one argument (no : or ). I think they are old ipsets. Are there ipsets with ?
I actually forgot to mention that traffic does not work. It is not just flooded logs and printout from that is the problem. Below is printout from in an ipv6-only cluster. As you can see all set's that have an "ip" have "family inet". Those without "ip", for instance "bitmap:port" has no family at all.
And I hope to send a PR for this in a few days.
I think the changes you made won't work. The problem is failing to create new ipsets. It should work after deleting the old ipsets.
They do work, I have tested. There are no old ipset's since the cluster is started fresh with an ipv6-only configuration. Do you upgrade your cluster from ipv4-only to ipv6-only without rebooting the nodes? That is a very complex upgrade and IMHO it should not be supported. When dual-stack comes I think it will be easier, like; to k8s with dual-stack support while still using ipv4-only /enable ipv6
I think I understand what you mean by "existing" sets now. Some sets are created twice since there are no clause in the if-statement. The ipset command is called with which I think causes the second "create" to actually become a "modify". And in that call the ipIpv6 flag is hard-coded to . I see no purpose for the isIpv6 flag in the structure and will remove it in my PR.


Question: | xargs -n1 dirname                                           | LC_ALL=C sort -u                                            ) DEFAULTER_FILES := $(addsuffix /$(DEFAULTER_FILENAME), $(DEFAULTER_DIRS)) <ins> DEFAULTER_EXTRA_PEER_PKGS :=  $(addprefix $(PRJ_SRC_PATH)/, $(DEFAULTER_DIRS)) </ins> # Reset the list of packages that need generation. $(shell mkdir -p $$(dirname $(META_DIR)/$(DEFAULTER_GEN)))

In the codegenerator we have: So basically we process only packages that are explicitly requested by the user and ignore ones that are imported. So in order to have the conversion func imported by the generator, the package where it lives has to be explicitly imported. So we should basically always pass all input packages to conversion generator. This is problematic, when regenerating conversions, we simply pass to input directories only those that were changed in the meantime: We should fix that - probably by passing all CONVERSIONDIRS as inputs to conversion-gen /kind-bug
I think this caused Not sure if was related yet.
It's the other way around - this is the main bug and root cause and the other that you linked is a consequence.
Also - FYI
I actually already started looking into it - so let me assign this.
is out for review
/triage accepted


Question: } for k, v := range tests { <del> got := MoreImportantPod(v.p1, v.p2) if got != v.expected { t.Errorf("%s failed, expected %t but got %t", k, v.expected, got) } </del> <ins> t.Run(k, func(t *testing.T) { got := MoreImportantPod(v.p1, v.p2) if got != v.expected { t.Errorf("expected %t but got %t", v.expected, got) } }) </ins> } }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently some unit tests in pkg/scheduler is not organized as Use subtests can make it easier to find out which case fails and leverage benefit of parallelism . Why is this needed: Follow Go test practice.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind feature /kind cleanup /sig scheduling /assign


Question: } } <ins> func TestSetDefaultReplicationControllerImagePullPolicy(t *testing.T) { containersWithoutPullPolicy, _ := json.Marshal([]map[string]interface{}{ { "name":  "install", "image": "busybox:latest", }, }) containersWithPullPolicy, _ := json.Marshal([]map[string]interface{}{ { "name":            "install", "imagePullPolicy": "IfNotPresent", }, }) tests := []struct { rc               versioned.ReplicationController expectPullPolicy versioned.PullPolicy }{ { rc: versioned.ReplicationController{ Spec: versioned.ReplicationControllerSpec{ Template: &versioned.PodTemplateSpec{ ObjectMeta: versioned.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": string(containersWithoutPullPolicy), }, }, }, }, }, expectPullPolicy: versioned.PullAlways, }, { rc: versioned.ReplicationController{ Spec: versioned.ReplicationControllerSpec{ Template: &versioned.PodTemplateSpec{ ObjectMeta: versioned.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": string(containersWithPullPolicy), }, }, }, }, }, expectPullPolicy: versioned.PullIfNotPresent, }, } for _, test := range tests { rc := &test.rc obj2 := roundTrip(t, runtime.Object(rc)) rc2, ok := obj2.(*versioned.ReplicationController) if !ok { t.Errorf("unexpected object: %v", rc2) t.FailNow() } if test.expectPullPolicy != rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy { t.Errorf("expected ImagePullPolicy: %s, got: %s", test.expectPullPolicy, rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy, ) } } } </ins> func TestSetDefaultService(t *testing.T) { svc := &versioned.Service{} obj2 := roundTrip(t, runtime.Object(svc))

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): initContainers imagePullPolicy initContainers required value Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: cluster/kube- Others: What happened: When trying to create either a ReplicaSet or Replication Controller whose Pod template contains an Init Container, the validation fails saying is a required value. What you expected to happen: The create should succeed and the Init Container should receive a default . How to reproduce it (as minimally and precisely as possible): Try to create either of the following: ReplicaSet Replication Controller Anything else do we need to know: If I add to the Init Container in the ReplicaSet or Replication Controller Pod template, the create succeeds. When I create an Init Container without in a StatefulSet Pod template, it succeeds. When I create an Init Container without in a bare Pod, it succeeds:
Looks like defaulting not being applied on server? That is probably due to removing defaulting from convert. Surprised we didn't catch this earlier
Some values still not being defaulted
, I'm going to create a PR if none works on it.
We decode from bytes, set defaults and then convert; but for init containers, we delay the decode from "decode phase" to "convert phase" at .

The defaults has been handled for initContainer at . /cc I'm working to find a solution for that; if any suggestion, please let me know.
Would this bug affect the deletion of jobs with an init-container? I ran into this bug and adjusted my manifest to add the pull policy to the init-container. However, I now cannot delete said job claiming a field is immutable. Example:
, not sure now. Let me check it,
Same error here with version 1.5.1, 1.5.2 The Job "grafana-import-dashboards" is invalid: [0].imagePullPolicy: Required value With this setup: kubectl apply -f monitoring Not working even if I specify a pullpolicy in the yaml. If I roll back to 1.4.7, it is working. Thank you. Greg.
any updates on a PR to address this issue? Is anyone from able to pick this issue up otherwise?
, here's the latest status: there's is a PR to generate for each Object (), it works for Object (Pod, Job) now but List; I'm working to make it generating for List.
Why would you need List to work? No client side logic should be decoding lists except as a generic submission.
The List tests are failed in . There's no for List, so no code to decode init-container annotation, e.g. of should call 's recursively.
Thanks for the update
+1
Any update on this? I still getting this issue with the last version of 1.5.3 on GKE?
ping for the comments? should we merge the PR for 1.5?
, there's a PR () that handled that issue; but we're discussing not to merge it because initContainers had been moved to fields, and fields is the recommended way to use since 1.6. But I'd like to get & 's comments on 1.5.
This is still an issue in 1.5.3 when using annotations: Without the deployment would not be accepted
ping & :).
i dont think this is fixed in 1.5 I just encountered this bug on 1.5.3
, yes, the PR is NOT merged to any branch, so the issue is still here. I'll create a PR for 1.5.
Hi Anyone help this issue.

#cat mongo- apiVersion: v1 kind: Pod metadata: labels: name: mongo-svc-a name: mongo-deployment spec: replicas: 1 template: metadata: labels: name: mongo spec: containers: image: mongo name: mongo ports: name: mongo containerPort: hostPort: volumeMounts: mountPath: "/data/db" name: mongopv volumes: name: mongopv persistentVolumeClaim: claimName: mongopvc When i want create mongo db container by using this file folowing error is occured. The Pod "mongo-deployment" is invalid:spec.containers: Required value
, it seems there're two here, please remove the first one and have a try.
Hi my kubectl binary is configured at ubuntu 16.04 machine.

NAME LABELS STATUS 10.1 Ready,SchedulingDisabled 10.2 10.1.10.2 Ready 10.3 Ready apiVersion: v1 kind: Node metadata: creationTimestamp: 2017-03-03T14:41:52Z labels: 10.1.10.1 name: 10.1.10.1 resourceVersion: "" selfLink: /api/v1/nodes/10.1.10.1 uid: -001f-11e7-8e83- spec: externalID: 10.1.10.1 status: addresses: address: 10.1.10.1 type: LegacyHostIP address: 10.1.10.1 type: InternalIP capacity: cpu: "2" memory: 4041352Ki pods: "110" conditions: lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-03-03T14:41:52Z message: kubelet has sufficient disk space available reason: KubeletHasSufficientDisk status: "False" type: OutOfDisk lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-04-05T15:27:52Z message: kubelet is posting ready status reason: KubeletReady status: "True" type: Ready nodeInfo: bootID: -6124-4ac7-ab96- containerRuntimeVersion: docker://1.12.6 kernelVersion: 4.9.16-coreos-r1 kubeProxyVersion: v1.2.4+coreos.1 kubeletVersion: v1.2.4+coreos.1 machineID: osImage: Container Linux by CoreOS 1298.7.0 (Ladybug) systemUUID: C3FD9852-2DFB-21F4-CE07-2597A746D99D When i create pv for creation pod is shows pv is pending. Please give any suggestions.....
Is there solution of this problem with version 2.7. Still facing the same. what exactly expectation of helm upgrade --install relname chartsdirectory is is expected to work on first time installation in case with CI/CD Pipelines.


Question: // ListedPaths returns the registered handler exposedPaths. func (m *PathRecorderMux) ListedPaths() []string { <ins> m.lock.Lock() </ins> handledPaths := append([]string{}, m.exposedPaths...) <del> sort.Strings(handledPaths) </del> <ins> m.lock.Unlock() sort.Strings(handledPaths) </ins> return handledPaths }

I ran pull-kubernetes-verify with race detection enabled (see ). It with: No errors. Run the integration test with - but I am not sure whether it triggers reliable outside of the Prow job. No response master n/a No response <details</details<details</details<details</details
/sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig apimachinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
Took a peek at this. Looks like writes to under a lock: But copies from it without a lock: This code is pretty old, I'm surprised we haen't caught this before
Well, We need a lock here..., I will file a fix for it. /assign
/triage accepted


Question: informerFactory := informers.NewSharedInformerFactory(client, controller.NoResyncPeriodFunc()) spc := newFakeStatefulPodControl(informerFactory.Core().V1().Pods(), informerFactory.Apps().V1beta1().StatefulSets()) ssu := newFakeStatefulSetStatusUpdater(informerFactory.Apps().V1beta1().StatefulSets()) <del> ssc := NewDefaultStatefulSetControl(spc, ssu, history.NewFakeHistory(informerFactory.Apps().V1beta1().ControllerRevisions())) </del> <ins> recorder := record.NewFakeRecorder(10) ssc := NewDefaultStatefulSetControl(spc, ssu, history.NewFakeHistory(informerFactory.Apps().V1beta1().ControllerRevisions()), recorder) </ins> stop := make(chan struct{}) informerFactory.Start(stop)

An issue to make bot happy for When StatefulSet finds any of its pods in phase Failed it will delete it and create it again. While doing so it will emit delete+create event which is really confusing for the user as he has no idea what's happening. /sig apps


Question: // Check pulls images required by kubeadm. This is a mutating check func (ipc ImagePullCheck) Check() (warnings, errorList []error) { <ins> policy := ipc.imagePullPolicy if len(policy) == 0 { policy = v1.PullIfNotPresent // Default behavior if the policy is unset } klog.V(1).Infof("using image pull policy: %s", policy) </ins> for _, image := range ipc.imageList { <del> switch ipc.imagePullPolicy { </del> <ins> switch policy { </ins> case v1.PullNever: <ins> klog.V(1).Infof("skipping pull of image: %s", image) </ins> continue case v1.PullIfNotPresent: ret, err := ipc.runtime.ImageExists(image)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --https://k8s- kubeadm-init 06-22 https://k8s- all kubeadm jobs are failing against k/k master. is this failing after we merged?
cc /sig cluster-lifecycle /area kubeadm kinder cluster preparation constructs images locally by downloading tarballs and tagging them with: then running it would find the image locally and not pull it from GCR. does not contain pre-release images, so trying to pull these images would fail. i think the problem here is that we broke the default behavior with
ok, the problem is indeed here: empty string "" should be the same as PullIfNotPresent.
should be fixed by
/triage accepted


Question: fp.syncProxyRules() // test cleanup left over <del> if CleanupLeftovers(ipvs, ipt, ipset, true) { </del> <ins> if CleanupLeftovers(ipvs, ipt, ipset) { </ins> t.Errorf("Cleanup leftovers failed") } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-proxy don’t clean ipvs rules with . What you expected to happen: kube-proxy clean ipvs rules with by default. How to reproduce it (as minimally and precisely as possible): Run kube-proxy in ipvs mode, and clean rules with . Anything else we need to know?: Based on , should cleanup all rules by default, including ipvs rules. And also retains functionality. Environment: Kubernetes version (use ): v1.20.0-bate.0 Cloud provider or hardware configuration: OS (e.g: ): centos7 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network
it seems that this flag entered deprecation, so I think that recovering functionaly is not the right approach you were involved in the discussion with this flag in , can you take a look at this?
thanks for your reply. The was in , to avoid removing IPVS rules which are not created by kube-proxy. According to , It seems that deleting ipvs rules by default is expected, but it is not so. I think that keeping --cleanup-ipvs is meaningful, unless kube-proxy can only delete the IPVS rules created by itself.
/assign
sorr for the late answer , but the flag must have the same functionality than , actually I can see that it has implemented. In case that the is not working we should fix it, but not recover the deprecated flag
Agree
sorry if I'm missing something. I just try to make the function of as described, not to recover the flag. The problem I see is: The can't flush IPVS. Because the IPVS flush process is still controlled by . Beforce , it was assigned by flag, but now it is not assigned, always false. currently does not have any functionality, just stored by . So, should we keep the function of temporarily by assigning it to ,or just make always flushing IPVS regardless of the ?

I prefer the former because the flag is not completely abandoned :)
it seems that is what the flag description says, you have to use to flush the ipvs entries if that is the case it seems that we are ok, right? no changes needed
In fact, can't flush ipvs entries. The reason is what I described above.
I´m sorry, I think I got it know, the flag is not exposed, but the option is there and initialized to true, this is what I see so, it seems that flush IPVS by default
yes, should flush IPVS by default, but it doesn’t actually
... and finally I got it, so sorry :bow: the ProxyServer is not being instantiated with the flag, despite is set :facepalm: you are more familiar with the IPVS implementation than me, I personally think that, if the flag deprecation has started, we should make that always flush IPVS and totally remove ... it seems aligned with the other proxy implementations
/triage accepted
Actually, after doing some code archaeology, this was the expectation per And the flag clearly mentions that --cleanup must flush IPVS entries too I think that we can go ahead and remove the --cleanup-ipvs flag and make --cleanup to flush ipvs too, right now is a mess and is not working at all , are you going to submit a PR?
thanks for digging up the history on this. That lines up with my recollection and the KEP around kube-proxy's clean up logic
OK, I will create a PR to remove the flag


Question: }, }, expectedDevicePath:      "/dev/rbd1", <del> expectedDeviceMountPath: fmt.Sprintf("%s/plugins/kubernetes.io/rbd/rbd/pool2-image-image2", tmpDir), </del> <ins> expectedDeviceMountPath: fmt.Sprintf("%s/plugins/kubernetes.io/rbd/mounts/pool2-image-image2", tmpDir), </ins> expectedPodMountPath:    fmt.Sprintf("%s/pods/%s/volumes/kubernetes.io~rbd/vol2", tmpDir, podUID), })

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: The method for ceph rbd volume plugin generates the path with code as follows: So the device mount path's format will be: , this implementation has two problems: with other volume plugins, as all the other plugins return , see as an example. method of ceph rbd plugin is not working properly, because rbd plugin uses method to find the device mount path(just like all the other plugins do), and method assumes the device mount path's format is (see ). What you expected to happen: The device mount path of rbd plugin should be . Anything else we need to know?: Do we have any reason for this design? If not, I will submit a PR to fix this, the adjusted method maybe look like this: /sig storage
/assign
nice catch There are two ways to fix this: Change mount path. But this will introduce compatibility issues during upgrade. Use , instead of to replace PR welcome
cc
Hi , there is a difference between and : returns the device path, such as , and returns the device mounted path, such as . What we really want is the second one, as we need to find the pool and image name. So we can't just simply replace with . Maybe we can find the mounted path specially for rbd plugin, use , the code looks like this:
thanks let make a forward moving plan: fix and append to the global path keep in . If no path is found, fall back to old path scheme using your code
Nice idea , I've submit a PR for this, PTAL.
I've checked and the reason is that I tested with the old release branch, the has been changed with in mount path, and causes the problem. thanks for your PR
i read again. There is another return path . So in rbd, the missing dir will not make into , but the second return should still give us the dir we need
You are right , indeed has two return path, but the second return still can't meet our needs. Because the parameter of is the mount path for a specific pod, such as , its format is , so will return the volume name. This volume name is returned by (see ), usually is the PV's name or the name specified in , both of these are not .


Question: } // Use a discovery client capable of being refreshed. <del> discoveryClient := rootClientBuilder.ClientOrDie("controller-discovery") cachedClient := cacheddiscovery.NewMemCacheClient(discoveryClient.Discovery()) </del> <ins> discoveryClient := rootClientBuilder.DiscoveryClientOrDie("controller-discovery") cachedClient := cacheddiscovery.NewMemCacheClient(discoveryClient) </ins> restMapper := restmapper.NewDeferredDiscoveryRESTMapper(cachedClient) go wait.Until(func() { restMapper.Reset()

Ran kube-controller-manager with and observed this in the log every 30 seconds: This is also happening in 1.20, 1.19, and 1.18 The GC and quota clients look up discovery information every 30 seconds. They have a QPS of 20 requests per second and a burst of 30 requests. Default configuration of kube-controller-manager runs without constant throttling warnings. /sig api-machinery
/assign /triage accepted


Question: } if len(pod.Spec.PriorityClassName) == 0 { var err error <del> priority, err = p.getDefaultPriority() </del> <ins> var pcName string pcName, priority, err = p.getDefaultPriority() </ins> if err != nil { return fmt.Errorf("failed to get default priority class: %v", err) } <ins> pod.Spec.PriorityClassName = pcName </ins> } else { pcName := pod.Spec.PriorityClassName if !priorityClassPermittedInNamespace(pcName, a.GetNamespace()) {

Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup /sig scheduling Description: If Pod did not set , admission controller will set to "default value". If the default value are from default priority class, it's better to also set of pod.
can I take this? /assign
, thanks :) I created a PR for functionality, but that'll be great if you can help add some e2e test for it. I can help to shepherd :)
thanks let me have a try :)


Question: } func TestReadRotatedLog(t *testing.T) { <del> if goruntime.GOOS == "windows" { // TODO: remove skip once the failing test has been fixed. t.Skip("Skip failing test on Windows.") } </del> tmpDir := t.TempDir() file, err := os.CreateTemp(tmpDir, "logfile") if err != nil {

The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows


Question: cmd.Flags().String("service-overrides", "", "An inline JSON override for the generated service object. If this is non-empty, it is used to override the generated object. Requires that the object supply a valid apiVersion field.  Only used if --expose is true.") } <del> func Run(f *cmdutil.Factory, cmdIn io.Reader, cmdOut, cmdErr io.Writer, cmd *cobra.Command, args []string) error { </del> <ins> func Run(f *cmdutil.Factory, cmdIn io.Reader, cmdOut, cmdErr io.Writer, cmd *cobra.Command, args []string, argsLenAtDash int) error { </ins> if len(os.Args) > 1 && os.Args[1] == "run-container" { printDeprecationWarning("run", "run-container") } <del> if len(args) == 0 { </del> <ins> // Let kubectl run follow rules for `--`, see #13004 issue if len(args) == 0 || argsLenAtDash == 0 { </ins> return cmdutil.UsageError(cmd, "NAME is required for run") }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We are removing CSI 0.3 support in 1.18, which is before selflink is going to be removed in 1.20. So we should just disable hostpath-v0 from alpha tests. /assign
/milestone v1.16


Question: return nil } <ins> // CreateOrRetainDeployment creates a Deployment if the target resource doesn't exist. If the resource exists already, this function will retain the resource instead. func CreateOrRetainDeployment(client clientset.Interface, deploy *apps.Deployment, deployName string) error { if _, err := client.AppsV1().Deployments(deploy.ObjectMeta.Namespace).Get(deployName, metav1.GetOptions{}); err != nil { if !apierrors.IsNotFound(err) { return nil } if _, err := client.AppsV1().Deployments(deploy.ObjectMeta.Namespace).Create(deploy); err != nil { if !apierrors.IsAlreadyExists(err) { return errors.Wrap(err, "unable to create deployment") } } } return nil } </ins> // CreateOrUpdateDaemonSet creates a DaemonSet if the target resource doesn't exist. If the resource exists already, this function will update the resource instead. func CreateOrUpdateDaemonSet(client clientset.Interface, ds *apps.DaemonSet) error { if _, err := client.AppsV1().DaemonSets(ds.ObjectMeta.Namespace).Create(ds); err != nil {

What happened: Upgrade using kubeadm failed because of CoreDNS migration enchancement What you expected to happen: Upgrade passed correctly How to reproduce it (as minimally and precisely as possible): any 1.15.x Kubernetes with CoreDNS that deployed with any unusual image (self built for example). We using commits counter as part of image tag, e.g - 1.6.2-100. to upgrade to 1.16.x with kubeadm. Anything else we need to know?: Logs part without ignoring preflight: Logs part with ignoring preflight: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OpenStack OS (e.g: ):Kernel (e.g. ):Install tools: Kubeadm
/sig cluster-lifecycle
The error came from It seems you should mention this issue on coredns project
True. Will notify them also.
/cc
/assign In such cases where the migration isn't supported and the user chooses to ignore the warnings from the preflight check, the migration should be skipped. wdyt? /cc
Yes. IMO, the warnings should be phrased like "CoreDNS configuration migration will not occur: reason"... so that a user expects the config migration will not be done.
/area kubeadm
SGTM
Upgrade passed correctly I think the best we can do here is to allow the upgrade to pass, but this will, in most practical cases, result in a cluster without a functioning DNS. The most common practical use-case for using a self built image of CoreDNS is to use external plugins. During the kubeadm upgrade, CoreDNS will be replaced with a stock release version of CoreDNS. When that new version of CoreDNS starts, it will not be able to load the existing Corefile's external plugins, and exit. ... or, we could finish the upgrade, leaving the existing CoreDNS deployment untouched, which I think works better.
Yea, it's kinda strong words that "Upgrade passed correctly". Probably, I think the best solution will be to make possible skip Corefile migration and make kubeadm finish the upgrade without Corefile changes. Guys that using custom images of CoreDNS should care about Corefile for new version by themselves, IMO.
Yes, I think that would work best...


Question: # This is where the final release artifacts are created locally <ins> readonly RELEASE_STAGE="${LOCAL_OUTPUT_ROOT}/release-stage" </ins> readonly RELEASE_DIR="${LOCAL_OUTPUT_ROOT}/release-tars" # ---------------------------------------------------------------------------

Sometimes it is nice to have non-tar'd binaries for releases. We should upload those along with the our packaged tarballs. This will help fix


Question: return nil } <del> func (ts *azureTokenSource) refreshToken(token *azureToken) (*azureToken, error) { </del> <ins> func (ts *azureTokenSource) Refresh(token *azureToken) (*azureToken, error) { return ts.source.Refresh(token) } // refresh outdated token with adal. // adal.RefreshTokenError will be returned if error occur during refreshing. func (ts *azureTokenSourceDeviceCode) Refresh(token *azureToken) (*azureToken, error) { </ins> env, err := azure.EnvironmentFromName(token.environment) if err != nil { return nil, err

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, when AAD login is configured, refresh token timeout would make kubectl print an error, not able to re-login without manually remove the tokens. Expect kubectl to promote user to login again. Why is this needed: Configure refresh-token timeout from Azure is a way to force re-login after inactivity. kubectl doesn't promote re-login makes users harder to apply timeout policy.
/sig azure
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig platform/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure
the root cause of this issue is in this first gets the cached token (result of device code flow) at it's trying to refresh an expired token at and failed. An error is returned. Since the expired token is still in the cache, it will always hit this path until user removes the token from kubeconfig. the fix should be: the token from cache a retry to re-initiate the device code flow in
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig platform/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
I'm working on it.
are you working on it? I will grab it if I dont hear it from you :)
have the code ready will try PR this weekend Best, Ping
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Closing as merged


Question: func NewTemplatePrinter(tmpl []byte) (*TemplatePrinter, error) { t, err := template.New("output"). <del> Funcs(template.FuncMap{"exists": exists}). </del> <ins> Funcs(template.FuncMap{ "exists":       exists, "base64decode": base64decode, }). </ins> Parse(string(tmpl)) if err != nil { return nil, err

Currently if one wants to view secrets stored in k8s they must follow the process described here: This involves copy&pasting. It seems to me that kubectl could provide a more helpful interface to base64 decrypt the users on behalf of the user saving them the trouble.
The kubelet does decode secrets before exposing them to the pod. Also the title doesn't really make sense. Secrets aren't encrypted so there's nothing to decrypt.
/kind feature
Yes, though this is low priority since we mostly don't want end users to view these once they get into the system.
Why can't that be an API permissions issue? I mean, if I really want to see it, I can just launch a pod to tell me.
I've update the issue title with what I understand as the request. To the best of my understanding this is just a request for us to avoid the need to pipe into base64 when inspecting secrets.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
I would like to try to make this work before it rots. cc Any guidance on a preferred implementation?
What do you think about adding support for functions in templates? This would allow someone to use the existing and a template that uses the function -- or a wide variety of other interesting functions as a side effect. I believe (but have not yet tested) that the following might be sufficient to enable the requested behaviour (plus the change to the dependencies list, of course):
Hi I suppose my thoughts on this issue have changed a little since I wrote it. The situation isn't quite as bad as I described as it is possible to decode a secret in one go without then need to copy and paste. i.e.

That said, that is a bit of a mouthful. Also, it might be nice to more easily 'work' with individual secrets and set them as well as get them. Should a user really need to pull down an entire group of secrets and edit a file just to be able to update a single password? For example something like this might be possible: What do people think? PS: Should this have been an issue on [kubectl] instead? PPS: I think some of the ugliness of this comes from configmaps being very similar to secrets except for the base64 encoding. Have there been any thoughts around bringing them into line? If memory serves, it might actually be nicer, due to YAML syntax-shenanigans, if configmaps were base64 encoded too.) [kubectl]:
Hi ... yes, I was thinking that it would be handy to be able to do something like: which is essentially the same as what you did without the overhead of the base64 process, or embed the information in a larger template like or in theory be able to use any of the other Sprig functions if someone found them useful. I think that having the Sprig methods available for generic formatting would just be useful on its own, really. On the topic of a or a , that seems pretty cool. There seem to be quite a lot of options for that I think people might expect to see mirrored in a if it were to exist. Thoughts?
It's also come to my attention that works great on MacOS but on Linux it's , so if someone, say, has a Helm chart where the notes tell the user to retrieve the secret data using one or the other of these, it won't work on the other one. would clearly be a simpler experience for the user than Hmm... thinking further: does does the idea conflict with the documentation in ? Perhaps we'd have to do something like: or similar?
/remove-lifecycle rotten
Small tip: you can use which works on both Linux and Mac.
Thank you
Update: I've looked at the huge amount of additional code that would be imported as a result of importing sprig and decided to go with a much smaller change instead, which is simply to implement a template function using . See PR with code and test.
please see the linked PR which addresses the original issue in a minimal fashion -- it simply provides a template function.

It has passed all the tests and needs review; if it's good enough to solve the original problem I'm happy to squash the commits and can hope to get it merged. As far as solving evolved description of the problem, I'm not sure where to start or if you really want me to go in that direction. Would you consider taking this change and splitting the additional work for being able to get / set the contents of an individual key in a secret into a separate issue? If you have any guidance on what you'd like to see with that I'd be happy to take a look, but it might be out of my current grasp.


Question: "context" "fmt" "os" <del> "os/exec" </del> "path/filepath" <del> "strconv" </del> "time" "k8s.io/apimachinery/pkg/fields"

xref: Graceful node shutdown test uses to invoke dbus - As per some distros, don't have installed so the test must be skipped in those environments. We should consider if we can switch to use or perhaps and fall back to to ensure the test can work across more distros.
/sig node
/sig testing
/remove-kind bug /kind feature I'm putting kind feature to signify that nothing is really broken in k8s, tests needs improvement.
/remove-sig testing This dbus command should be up to SIG node, though tentatively: /triage accepted
/area test
Do we actually have these other distros available to prevent regressions if we change that?
/assign
the change will be to replace skip logic when is missing to use an alternative way to send this. So there will be no regression on distros where it currently runs without the skip. Original bug should have a link to job that was failing
I would like to work on improving the command. Am I understanding correct here We have to change to logic where we skip to use busctl instead. Could you please give me some idea on how can we make it to use , some similar examples in the repo would be really useful.
please have a look at The issue is that the command available is too old and doesn't include .
I see, it looks like we are not just adding a condition for instead we are replacing it with . It looks you are already working on it. Please inform if I can help in anyway. Thanks for the response.
Both are possible... it was easier to get a CI signal if that was used in all cases. I might put the condition later. I am waiting for comments in the PR, but feel free to propose other solutions that might work with older systemd versions.


Question: <del> 2.25 </del> <ins> 2.26 </ins>

<!-- STOP -- PLEASE READ! GitHub is not the right place for support requests. If you're looking for help, check the or our . You can also post your question on the or the forum. If the matter is security related, please disclose it privately via --s390x support for nonewprivs image was in and the commit from was cherry picked in to fix agnhost broken on s390x. However, the post merge job failed for both( ) as image builder was broken. Now that image builder has been fixed in , these jobs need to be restarted to push s390x tags of agnhost and nonewprivs in staging bucket. cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
as you are on call this week, can you please run the CI job mentioned by hand?
, when I try to run the job with , it failed with . So I went over and looked at the history and found that this job was deleted by on 11/11. I don't think prow allows running a job that doesn't exist in its configmap, and searching the prow history with got: This is the list of jobs that have ever ran since prow keeps all history in GCS, which indicates that not all of the new jobs in has ran yet, can you take a look at and let us know which job is that you want?
thanks for looking. let's ask what jobs need to be triggered.
Hello, The post-kubernetes-push-e2e-test-images has been split into individual jobs for individual images and are now triggered by changes made in that image's folder. The jobs haven't been run yet since the split. You can trigger a job by sending a PR (e.g.: bumping the VERSION) and getting it merged.
/reopen Still failing according should fix it.
Reopened this issue. <detailsIn response to : /reopen Still failing according should fix it. Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close I am closing this issue as the agnhost job passed and the multiarch image is now available, it works fine for s390x as conformance job passed with the same
Closing this issue. <detailsIn response to : /close I am closing this issue as the agnhost job passed and the multiarch image is now available, it works fine for s390x as conformance job passed with the same Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/kubernetes/pkg/apis/componentconfig" internalapi "k8s.io/kubernetes/pkg/kubelet/api" runtimeapi "k8s.io/kubernetes/pkg/kubelet/api/v1alpha1/runtime" <ins> kubecm "k8s.io/kubernetes/pkg/kubelet/cm" </ins> kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/dockershim/cm" "k8s.io/kubernetes/pkg/kubelet/dockertools"

Kubernetes version (use ): v1.5.0-alpha.1.2189+-dirty docker: 1.12.3 Environment: fedora24 docker-engine-1.12.3-1.fc24.x86_64 What happened: if docker Cgroup Driver=cgroupfs, and set --cgroup-driver=systemd in kubelet, then enable --experimental-cgroups-per-qos=true; The container cgroup some is not correct. How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: 1) docker Cgroup Driver=cgroupfs and --cgroup-driver=cgroupfs in kubelet works well 2) docker Cgroup Driver=systemd and --cgroup-driver=systemd in kubelet works well
help check, thanks.
In Kubernetes 1.6, we can work to detect this case and return an error. IMHO its a configuration error to have docker running with a cgroupfs driver paired with the kubelet running the systemd cgroup driver. /cc
We can also document the compatibility matrix for now.


Question: MountOptions:      []string{"foo"}, VolumeBindingMode: &modeImmediate, }, <ins> { TypeMeta: metav1.TypeMeta{ Kind: "StorageClass", }, ObjectMeta: metav1.ObjectMeta{ Name: "csi", }, Provisioner:       "mydriver.csi.k8s.io", Parameters:        class1Parameters, ReclaimPolicy:     &deleteReclaimPolicy, VolumeBindingMode: &modeImmediate, }, </ins> } // call to storageClass 1, returning an error

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In clusters that have intree drivers as the default storageclass, if you try to create a PVC with snapshot data source and forget to put the csi storageclass in it, then an empty PVC will be provisioned using the default storageclass. What you expected to happen: PVC creation should not proceed and instead have an event with an incompatible error, similar to how we check proper csi driver in the csi provisioner. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
I think similar issue exists with cloning. We'll need to think about what the behavior should be for data populators too. cc
/triage accepted
Scenario to reproduce the issue Create the nginx deployment and the PVC used by it Exec into the running container, cd into the PVC mounted path and create some files Create the following using as source the The used is the following and wait until it becomes Delete the deployment and the initial PVC Create a new PVC, using the previously created as its Set the name of the new (restored) PVC to the nginx deployment and create the again into the PVC mounted directory. It should contain the previously created files but its empty
can you modify the title of this issue and change to ? In addition to snapshot datasource, using PVC datasource and AnyVolumeDataSource should also fail for in-tree PVC.
/assign
I wasn't 100% sure on data populators. Because it shouldn't have a dependency on CSI right?
I guess even if data populators does support intree volumes, pv controller should ignore it similar to csi provisioner
It may or may not have a dependency on CSI. Design is still in progress. We can leave it out for now until we know exactly how to use it. One thing I know is that the any volume data source cannot be an in-tree k8s resource.
Yes, we can actually just ignore it in pv controller.

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm getting same with longhorn, is this issue resolved on your side?


Question: e.queue.Add(key) } <ins> func (e *EndpointController) onEndpointsDelete(obj interface{}) { key, err := controller.KeyFunc(obj) if err != nil { utilruntime.HandleError(fmt.Errorf("Couldn't get key for object %+v: %v", obj, err)) return } e.queue.Add(key) } </ins> // worker runs a worker thread that just dequeues items, processes them, and // marks them done. You may run as many of these in parallel as you wish; the // workqueue guarantees that they will not end up processing the same service

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? no. What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): service endpoints missing Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.6.2 Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Install tools: manual Others: What happened: Service in namespace sees no endpoints that are available What you expected to happen: For it to see available endpoints How to reproduce it (as minimally and precisely as possible): Not sure I can. It appeared on the latest remove/replace of worker nodes, but was fine before Anything else we need to know: I have a traefik Ingress controller defined and a service for it: Selectors match, the pods are running just fine, yet the endpoints do not show up. I suspect if I delete the service and recreate it, it probably will pick it up, but I shouldn't have to, and am concerned it might occur again; rather actually debug it. How can I check what kube is doing to affiliate it? How often does it check? Where?
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
The system is asking for a SIG, but it isn't clear which SIG is appropriate for this. Taking best guess, apologies if I am wrong.

/sig sig/apps /sig sig/scheduling
/sig network
can you provide the following output?
I wish I could... in the week since, the cluster has been deleted and recreated several times. AWS instances may be cheap, but still aren't free. Client didn't want them sticking around too long for the cost.
Should we close this issue pending more information or keep it open for someone to test out?
we are killing and recreating clusters with some regularity over these weeks. I am happy if you want to close it since I cannot recreate it yet. Just let me reopen it if I manage to recreate and get more info. Just tell me, please, if it does come up again, what additional data should I get? There isn't much data in the docs on how the service resolves what endpoints it should use and how to figure out why it is misbehaving when it does. Happy to take pointers.
Please reopen when it happens again. Please record related endpoints objects and pod objects.
I'm facing same issue with Nginx by trying to expose the nodePort for Nginx-ingress-controller pod
Try .
Your service is using while your pods are using .
Sorry, didn't notice that
OS: ubuntu 16.04 kubernetes version: v1.8.4 hi, I have the same problem now. $ kc get endpoints -njenkins This situation has been going on for a long time and no solution has been found. Help out. Thanks
Same the problem, svc lost the endpoint after a few time. If deleting the pod, then the svc endpoint is show up again. How to debug this problem?
Im having the same problem on Windows Nodes, after some hours all svc on one node loses their endpoints. Kubernetes V1.15
May be fixed by this?

same issue for a custom helm that I created: kubectl version Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-19T10:52:43Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-04-03T04:12:37Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} kubectl describe service/vnfdb-ui Name: vnfdb-ui Namespace: test-helm Labels: Annotations: <noneSelector: , Type: LoadBalancer IP: 10.105.168.158 Port: http 80/TCP TargetPort: http/TCP NodePort: http /TCP Endpoints: <noneSession Affinity: None External Traffic Policy: Cluster Events: <none$ kubectl get pods -l , NAME READY STATUS RESTARTS AGE vnfdb-ui--ttrwm 1/1 Running 0 9m $ kubectl get endpoints vnfdb-ui NAME ENDPOINTS AGE vnfdb-ui <none9m33s $ kubectl get endpoints vnfdb-ui -o yaml apiVersion: v1 kind: Endpoints metadata: creationTimestamp: "2019-08-24T21:28:58Z" labels: vnfdb Tiller ui "1.0" ui-0.1.0 name: vnfdb-ui namespace: test-helm resourceVersion: "" selfLink: /api/v1/namespaces/test-helm/endpoints/vnfdb-ui uid: -c6b6-11e9-a0d5- Any clue what is wrong here and how can I fix it?
Got it, apparently realized that its not only the selector that has to match labels. The port names should be the same in the pod definition and services definition as well. The latter was the culprit in my case.
Same the problem.My service and pod is running. But the service do not has endpoint,and this situation lasts for 10 hours. It is OK after I restart controller manager.

My last operation was to delete the service and create the service again. I have not done any other operations since then. Kubernetes V1.11.5
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Good catch!
Happens to us once in a while: We have a process creating a service, then a test deleting that same service, which ends up being automatically recreated. Once in a while (every ~100 test), the re-created service never gets any endpoints, even after waiting for several hours. I'm fairly confident recreating the service again fixes it, but that's not a solution. It just happened on a GKE cluster in version , for which I don't have controller-manager logs available unfortunately. I'll try to reach GCP support, they may be able to troubleshoot more from the controller-manager logs. Edit: the endpoint magically appeared after 22 hours.
Thanks for the detailed bug report! I'll try to recreate this scenario with some new tests. /assign
A quick follow up to let you know we haven't forgotten about this issue. From what we can tell, this is being caused by an Endpoints informer cache that can potentially be out of date when syncService() runs. Essentially syncService() would be called twice - once for the delete and once for the create event.

There's a chance that when the second run of syncService() happens the Endpoints cache may not have been updated with the latest run. If that happens, syncService() will not make any changes because it sees an endpoints resource that has all the right ports and addresses set. is working on a fix here that will mirror what we've done for the EndpointSlice controller with the . This will involve watching Endpoints changes and calling syncService() if the change has not already been accounted for by the controller. As far as I can tell, this should be a relatively rare bug. It should only happen if the cache is out of date, the recreated Service is identical, and the Pods have not changed at all. Given the relative complexity of the fix here, I'm not sure how far we'll be able to backport this. At the very least, we're hoping to have a fix in for 1.19.
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think is almost ready, sorry for not catching the lifecycle close earlier.
Dude, you saved my life.


Question: var _ = framework.KubeDescribe("DNS horizontal autoscaling", func() { f := framework.NewDefaultFramework("dns-autoscaling") var c clientset.Interface <del> var nodeCount int </del> var previousParams map[string]string <del> DNSParams_1 := map[string]string{"linear": "{"nodesPerReplica": 1}"} DNSParams_2 := map[string]string{"linear": "{"nodesPerReplica": 2}"} DNSParams_3 := map[string]string{"linear": "{"nodesPerReplica": 3}"} </del> <ins> DNSParams_1 := DNSParamsLinear{map[string]string{"linear": "{"nodesPerReplica": 1}"}, 1.0, 0.0} DNSParams_2 := DNSParamsLinear{map[string]string{"linear": "{"nodesPerReplica": 2}"}, 2.0, 0.0} DNSParams_3 := DNSParamsLinear{map[string]string{"linear": "{"nodesPerReplica": 3, "coresPerReplica": 3}"}, 3.0, 3.0} </ins> BeforeEach(func() { framework.SkipUnlessProviderIs("gce") c = f.ClientSet <del> nodes := framework.GetReadySchedulableNodesOrDie(c) nodeCount = len(nodes.Items) Expect(nodeCount).NotTo(BeZero()) </del> <ins> Expect(len(framework.GetReadySchedulableNodesOrDie(c).Items)).NotTo(BeZero()) </ins> pcm, err := fetchDNSScalingConfigMap(c) <del> ExpectNoError(err) </del> <ins> Expect(err).NotTo(HaveOccurred()) </ins> previousParams = pcm.Data By("Replace the dns autoscaling parameters with testing parameters") <del> ExpectNoError(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(DNSParams_1))) </del> <ins> Expect(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(DNSParams_1.data))).NotTo(HaveOccurred()) </ins> By("Wait for kube-dns scaled to expected number") <del> ExpectNoError(waitForDNSReplicasSatisfied(c, nodeCount, DNSdefaultTimeout)) </del> <ins> getExpectReplicasLinear := getExpectReplicasFuncLinear(c, &DNSParams_1) Expect(waitForDNSReplicasSatisfied(c, getExpectReplicasLinear, DNSdefaultTimeout)).NotTo(HaveOccurred()) </ins> }) AfterEach(func() { By("Restoring intial dns autoscaling parameters") <del> ExpectNoError(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(previousParams))) </del> <ins> Expect(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(previousParams))).NotTo(HaveOccurred()) </ins> }) // This test is separated because it is slow and need to run serially

https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal

autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
should this issue be moved into the 1.5 milestone and is it a release blocker? The test seems to flake fairly infrequently but we need to get the issue triaged as to whether it's a 1.5 release blocker. Please update this issue by 1PM PST 30 November 2016. Thanks! cc:
Since the feature it is testing is introduced for 1.5, so it should have the 1.5 milestone? This test is flaking because it is not well written, hence I don't think it is a release blocker. And I already sent out a which got LGTM before but recently rebased.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
Thanks for triaging
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}