Question: azureSource := volume.AzureDisk pv := &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ <del> // A.K.A InnerVolumeSpecName required to match for Unmount Name: volume.Name, </del> <ins> // Must be unique per disk as it is used as the unique part of the // staging path Name: fmt.Sprintf("%s-%s", AzureDiskDriverName, azureSource.DiskName), </ins> }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{

Minimal Repro: Create 2 different disks Create 2 Pods each using one of the disks Start both pods on the same node You will notice one of the disks gets ed to a path: , then both disks are ed from to a target path: which is similar but at least uses the pod . Both of these volumes are being treated as the same volume in the Kubelet even though they are definitely not the same volume. Note: it may not be "treated" as the same volume - it may just be that since the paths are the same the following volumes see that the path is already a mountpoint and skip the step. This is the more likely scenario based on observations. The value is the "name" of the inline-volume. These can obviously be the same across different pods - we should actually be placing it in a more unique directory (based off the translated maybe?). /cc
/sig storage
The inline volume example I'm using (ignore the fact that is a complete misnomer): I've confirmed that changing the "name" makes inline volumes work well since there is no longer a collision on the paths
/assign
Alright, problem is sort of twofold: The referred to above for inline-volumes is set here: As we can see from the comment: it is set to the because the in-tree CSI Plugin makes a lot of naming assumptions based on the which for PVC/PVs is the PV name (unique in a cluster). Primarily, during unmount the CSI Plugin tries to from a "well known ": which happens to be some path based on the which for the inline volume is just the ( above). So we run into the problem that if we use the directory is not sufficiently unique for different disks. And if we use something else like then the unmount pathway is unable to find the . I think using is a no-go. So we need to use something sufficiently unique (like ) for the ( above) and fix up some of the other codepaths to either use different "well-known paths" to write to, or to fix up path generation code so they give us the correct locations. I'm also going to check to see whether 's refactor fixes this issue (because it might since it translates things earlier)
OK. Really good news.

your refactor in its current state actually fixes this issue if we set the to something unique here: Looks like the early translation is also making the pathway from the "correct" dir for inline migration. So I will create a PR that fixes this issue that should merge after


Question: ResourceTypeOrNameArgs(args...). Flatten(). Do() <ins> checkErr(r.Err()) </ins> found := 0 err = r.IgnoreErrors(errors.IsNotFound).Visit(func(r *resource.Info) error {

I've reported this before.
From a Makefile I get exit status 255.
Just to be clear (sorry for the basic question), are you saying that when the service file is malformed the crash is unhelpful for debugging? Or that creating any/all services using is currently broken?
The YAML file is not malformed. I think there was something perhaps not quite right about firewalls/rules etc. and this caused to crash unhelpfully.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What you expected to happen: if returns true I expect to be able to perform the action without issue. Note in this case my service user has a rolebinding in a single namespace (and that is also what my kubecontext is set to.) if I change my default to a different namespace then the command returns false How to reproduce it (as minimally and precisely as possible): create a namespace (test) create a service account in that namespace create a role that gives get list and watch permissions to create a rolebinding from that role to the service account. Anything else we need to know?: this is special because namespaces are not scoped to a namespace. if you replace the command with then you get the exepcted result however this breaks the path of least suprise. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig auth
asks the authorization system if the given verb can be performed on the given noun at the given scope. It does not interpret whether the resource exists at that scope. We could warn if we find the specified resource and it does not exist at the specified scope (just like we warn if you ask about a resource that we can't find), but we would still interrogate the authorization system and return the results as we do today. /kind feature /remove-kind bug
/cc


Question: cmds.AddCommand(NewCmdExplain(f, out)) cmds.AddCommand(NewCmdConvert(f, out)) <del> if cmds.Flag("namespace").Annotations == nil { cmds.Flag("namespace").Annotations = map[string][]string{} </del> <ins> if cmds.Flag("namespace") != nil { if cmds.Flag("namespace").Annotations == nil { cmds.Flag("namespace").Annotations = map[string][]string{} } cmds.Flag("namespace").Annotations[cobra.BashCompCustom] = append( cmds.Flag("namespace").Annotations[cobra.BashCompCustom], "__kubectl_get_namespaces", ) </ins> } <del> cmds.Flag("namespace").Annotations[cobra.BashCompCustom] = append( cmds.Flag("namespace").Annotations[cobra.BashCompCustom], "__kubectl_get_namespaces", ) </del> return cmds }

ci-kubernetes-e2e-gce-iscsi-serial and ci-kubernetes-e2e-gce-iscsi I guess it is create cluster step not tests Not clear I guess may be From I guess it is releated to /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
From I get I think there's a problem here.


Question: ## Changelog since v1.15.1 <del> **No notable changes for this release** </del> <ins> * Fix CVE-2019-11249: Incomplete fixes for CVE-2019-1002101 and CVE-2019-11246, kubectl cp potential directory traversal ([#80436](https://github.com/kubernetes/kubernetes/pull/80436)) * Fix CVE-2019-11247: API server allows access to custom resources via wrong scope ([#80750](https://github.com/kubernetes/kubernetes/pull/80750)) See also the [security announcement for this release](https://groups.google.com/forum/#!topic/kubernetes-security-announce/vUtEcSEY6SM). </ins> # v1.15.1

What would you like to be : The release notes state "No notable changes for this release": However it turns out these releases fix important security problems: Why is this needed: If people track releases in, for example, GitHub and then use the release notes to see what's changed, they will totally miss that important security problems have been fixed in these releases. People should not have to look in multiple places to find news of such fixes. The current practice risks people not upgrading because there were "no notable changes" and being oblivious to the risks they run.
I expect this is one for and probably others but I don't really have any clue who.
/sig release This has to do with the timing of cutting the releases vs sending the announcements. The notices were included in the announcement, but not in the generated md files. /assign can you open a PR to update those?
Looks like they put it in the email and the group, but not the release notes. !topic/kubernetes-security-announce/vUtEcSEY6SM/discussion
Just opened a PR to fix this: (I only noticed that this issue was already assigned after committing my changes, hope that's OK)
Thanks that PR looks exactly like what I'm after for this particular update!


Question: trap cleanup EXIT SIGINT <del> kube::golang::setup_env </del> TMP_DIR=${TMP_DIR:-$(kube::realpath "$(mktemp -d -t "$(basename "$0").XXXXXX")")} ETCD_HOST=${ETCD_HOST:-127.0.0.1} ETCD_PORT=${ETCD_PORT:-2379}

we just log that the user should install etcd, which is impractical in an ephemeral container we actually already call installing etcd in the verify script, but we don't export it into the current PATH for the process so it doesn't work ... ref: /sig api-machinery /assign /kind bug
/triage accepted
for one possible fix
/sig etcd


Question: } } <ins> // TestCloseAllRace ensures CloseAll works with connections being simultaneously dialed func TestCloseAllRace(t *testing.T) { conns := int64(0) dialer := NewDialer(func(ctx context.Context, network, address string) (net.Conn, error) { return closeOnlyConn{onClose: func() { atomic.AddInt64(&conns, -1) }}, nil }) done := make(chan struct{}) wg := &sync.WaitGroup{} // Close all as fast as we can wg.Add(1) go func() { defer wg.Done() for { select { case <-done: return default: dialer.CloseAll() } } }() // Dial as fast as we can wg.Add(1) go func() { defer wg.Done() for { select { case <-done: return default: if _, err := dialer.Dial("", ""); err != nil { t.Error(err) return } atomic.AddInt64(&conns, 1) } } }() // Soak to ensure no races time.Sleep(time.Second) // Signal completion close(done) // Wait for goroutines wg.Wait() // Ensure CloseAll ran after all dials dialer.CloseAll() // Expect all connections to close within 5 seconds for start := time.Now(); time.Now().Sub(start) < 5*time.Second; time.Sleep(10 * time.Millisecond) { // Ensure all connections were closed if c := atomic.LoadInt64(&conns); c == 0 { break } else { t.Logf("got %d open connections, want 0, will retry", c) } } // Ensure all connections were closed if c := atomic.LoadInt64(&conns); c != 0 { t.Fatalf("got %d open connections, want 0", c) } } </ins> type closeOnlyConn struct { net.Conn onClose func()

What happened: my kubelet restart unexpectedly,i find log info has more "Failed to list" info and has fatal error: go of nil func value,and then kubelet restart Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"logging"/"elasticsearch-token-99wc2": Failed to list v1.Secret: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] failed 5 attempts to update node lease, will retry after 10s Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"logging"/"default-token-6wh7n": Failed to list v1.Secret: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"monitoring"/"prometheus-rules": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"logging"/"service-registry": Failed to list v1.Secret: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"monitoring"/"prometheus-center": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:30 mgt15 kubelet[6914]: E0202 17:46:30. 6914 ] Failed to list v1beta1.RuntimeClass: Get https://vip- Feb 02 17:46:31 mgt15 kubelet[6914]: E0202 17:46:30. 6914 ] object-"logging"/"telegraf-elasticsearch": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:31 mgt15 kubelet[6914]: E0202 17:46:30.

6914 ] object-"monitoring"/"prometheus-center": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:32 mgt15 kubelet[6914]: fatal error: go of nil func value Feb 02 17:46:32 mgt15 kubelet[6914]: goroutine 357 [running]: Feb 02 17:46:32 mgt15 kubelet[6914]: (closableConn).Close(0x4002193f00, 0x0, 0x0) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- Feb 02 17:46:32 mgt15 kubelet[6914]: (Dialer).CloseAll(0x4000bbaea0) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- Feb 02 17:46:32 mgt15 kubelet[6914]: (Kubelet).updateNodeStatus(0x40010ce000, 0x54d0978, 0x40010ce648) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- +0xf0 Feb 02 17:46:32 mgt15 kubelet[6914]: (Kubelet).syncNodeStatus(0x40010ce000) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- What you expected to happen: kubelet do not restart How to reproduce it (as minimally and precisely as possible): when have more info of "Failed to list" Environment: Kubernetes version (use ):v1.14.3 OS (e.g: ):Ubuntu 18.04.3 LTS Kernel (e.g.

):4.15.0-55-generic Network plugin and version (if this is a network-related bug):calico3.8.2
/sig node /wg node /committee node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig node /wg node /committee node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fix is in
/sig auth
/assign
thankswhat version is this bug been fixed?
The fix is not yet merged, but I plan to backport the fix to 1.15.x, 1.16.x, 1.17.x (the exact patch versions depend on when the backports are merged)


Question: fmt.Printf("%#vn", version.Get()) os.Exit(0) } else if *versionFlag == VersionTrue { <del> fmt.Printf("Kubernetes %sn", version.Get()) </del> <ins> fmt.Printf("%s %sn", programName, version.Get()) </ins> os.Exit(0) } }

When using the package, one can easily add a flag to a CLI tool. However, the output of invocation with is hard-coded to . As such, the package is not very usable in non-Kubernetes projects. /wg component-standard /assign


Question: } podName := pods.Items[0].Name <del> queryDNS := fmt.Sprintf(queryDNSPythonTemplate, backendSvcName+"."+namespaces[0].Name) </del> <ins> queryDNS := fmt.Sprintf(queryDNSPythonTemplate, backendName+"."+namespaces[0].Name) </ins> _, err = framework.LookForStringInPodExec(namespaces[0].Name, podName, []string{"python", "-c", queryDNS}, "ok", dnsReadyTimeout) framework.ExpectNoError(err, "waiting for output from pod exec") <del> updatedPodYaml := prepareResourceWithReplacedString(frontendPodYaml, fmt.Sprintf("dns-backend.development.svc.%s", framework.TestContext.ClusterDNSDomain), fmt.Sprintf("dns-backend.%s.svc.%s", namespaces[0].Name, framework.TestContext.ClusterDNSDomain)) </del> <ins> updatedPodYaml := strings.Replace(read(filepath.Join(clusterDnsPath, "dns-frontend-pod.yaml")), fmt.Sprintf("dns-backend.development.svc.%s", framework.TestContext.ClusterDNSDomain), fmt.Sprintf("dns-backend.%s.svc.%s", namespaces[0].Name, framework.TestContext.ClusterDNSDomain), 1) </ins> // create a pod in each namespace for _, ns := range namespaces { <del> framework.NewKubectlCommand(ns.Name, "create", "-f", "-", getNsCmdFlag(ns)).WithStdinData(updatedPodYaml).ExecOrDie(ns.Name) </del> <ins> framework.RunKubectlOrDieInput(ns.Name, updatedPodYaml, "create", "-f", "-") </ins> } // wait until the pods have been scheduler, i.e. are not Pending anymore. Remember // that we cannot wait for the pods to be running because our pods terminate by themselves. for _, ns := range namespaces { <del> err := e2epod.WaitForPodNotPending(c, ns.Name, frontendPodName) </del> <ins> err := e2epod.WaitForPodNotPending(c, ns.Name, frontendName) </ins> framework.ExpectNoError(err) } // wait for pods to print their result for _, ns := range namespaces { <del> _, err := framework.LookForStringInLog(ns.Name, frontendPodName, frontendPodContainerName, podOutput, framework.PodStartTimeout) framework.ExpectNoError(err, "pod %s failed to print result in logs", frontendPodName) </del> <ins> _, err := framework.LookForStringInLog(ns.Name, frontendName, frontendName, podOutput, framework.PodStartTimeout) framework.ExpectNoError(err, "pod %s failed to print result in logs", frontendName) </ins> } }) }) <del> func getNsCmdFlag(ns *v1.Namespace) string { return fmt.Sprintf("--namespace=%v", ns.Name) } // pass enough context with the 'old' parameter so that it replaces what your really intended. func prepareResourceWithReplacedString(inputFile, old, new string) string { f, err := os.Open(inputFile) framework.ExpectNoError(err, "failed to open file: %s", inputFile) defer f.Close() data, err := io.ReadAll(f) framework.ExpectNoError(err, "failed to read from file: %s", inputFile) podYaml := strings.Replace(string(data), old, new, 1) return podYaml } </del> // waitForServiceResponding waits for the service to be responding. func waitForServiceResponding(c clientset.Interface, ns, name string) error { ginkgo.By(fmt.Sprintf("trying to dial the service %s.%s via the proxy", ns, name))

What happened: We've been doing some release readiness and auditing all tagged ginkgo tests... Looks like The test seems to expect a GOPATH when running. Im seeing some failures in sonobuoy and suspect a possible relation there but havent dove into it. Either way, its clear this test might need some updating/fixing/cleanuping and parts can be rewritten/improved. What you expected to happen: No reliance on GOPATH for an e2e test ... I dont remember the whole history on this, but no tests that rely on static files currently look in gopath, from what i can tell ? instead we plumbed in a repo-root/ option a few years ago. There may be a state of the art in terms of how we do this.... but i thought it was at one point which was an option passed in. How to reproduce it (as minimally and precisely as possible): Try to run this in sonobuoy or any other environment where there's no GOPATH or non-standard source trees, pretty sure it might not work. Anything else we need to know?: I think the fix here is to just embed the YAML in golang like we do in other tests. along the way there's other cleanup we can do to improve this test. Environment: Kubernetes version (use ): master
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
the YAML files are pretty small, so +1 to embed them in i don't think testing the YAML from the separate repository () directly is required here. ref /help /kind cleanup /priority backlog
/remove-help
/assign Also I should say, I talked to Jay on slack and he is ok with me taking a look at it, if thats ok.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle stale
/remove-lifecycle stale
xref: go 1.16 has built in support for embedding go 1.17 looks to be removing GOPATH
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/help wanted
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten still a problem. FYI re: GOPATH removal
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/assign


Question: detect-master() { KUBE_MASTER=${MASTER_NAME} <ins> echo "Waiting for ${MASTER_NAME} IP Address." echo echo "  This will continually check to see if the master node has an IP address." echo </ins> KUBE_MASTER_IP=$(nova show $KUBE_MASTER --minimal | grep accessIPv4 | awk '{print $4}') <ins> while [ "${KUBE_MASTER_IP-|}" == "|" ]; do KUBE_MASTER_IP=$(nova show $KUBE_MASTER --minimal | grep accessIPv4 | awk '{print $4}') printf "." sleep 2 done echo "${KUBE_MASTER} IP Address is ${KUBE_MASTER_IP}" </ins> } # $1 should be the network you would like to get an IP address for

More often than not kube-up will loop forever for cluster initialization. I eventually realized it was because the script was looping forever on "https://|/api/v1beta1/pods". This happens any time Rackspace doesn't very quickly assign an IP address to the master. PR incoming with pretty rudimentary looping on the code that does the IP address lookup.


Question: k8s.io/apimachinery/pkg/runtime/serializer/recognizer k8s.io/apimachinery/pkg/runtime/serializer/streaming k8s.io/apimachinery/pkg/runtime/serializer/versioning <ins> k8s.io/apimachinery/pkg/runtime/serializer/yaml </ins> k8s.io/apimachinery/pkg/selection k8s.io/apimachinery/pkg/types k8s.io/apimachinery/pkg/util/cache

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add an external MetaFactory that allows for serializing multiple YAML (or JSON) files in kubeadm. The logic of marshalling YAML should be factored out from to . Why is this needed: This will allow processing component configs containing multiple documents per file. Additionally, an out-of-tree library would allow sharing this functionality with other components.
I will try work on this as carry-on from /sig cluster-lifecycle /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: { Name:  strings.ToLower(name), Image: image, <del> Ports: portsFromString(portSpec), </del> <ins> Ports: ports, </ins> }, }, },

Kubernetes will create replication controller and pods in case of port spec error, e.g. will create 'myNginx'. A following correct spec will fail:
/cc


Question: # Verify prereqs on host machine function verify-prereqs() { # Check the OpenStack command-line clients <del> for client in swift glance nova openstack; </del> <ins> for client in swift glance nova heat openstack; </ins> do if which $client >/dev/null 2>&1; then echo "${client} client installed"

ci-kubernetes-e2e-gce-master-new-gci-kubectl-skew kubetest.SkewTest 2022-04-14 15:13:42 +0000 UTC The error message is It's complaining about "file contents" and not "url contents". And because this is a skew test I am assuming this is due to The fix might be to cherry pick this to 1.23. No response /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I've opened the cherry pick at if we decide that's the right route. The other options I can think of are to revert this change on master or to add the old environment variable back in as a file path. /assign /priority critical-urgent
Tracked the root of this to This should start passing once that merges. The cherry pick will also fix. We really need to get skew tests running on PRs.


Question: <del> 2.14 </del> <ins> 2.15 </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The image used in E2E tests is used for testing a wide variety of OS-agnostic features. One of these is , which runs an HTTP server with various endpoints that can be used to validate things like request/response. It would be useful to allow for 's functionality to also support HTTPS so all of that logic can be tested for TLS-based traffic. Currently, can be run like: I'm proposing something along the lines of: The default certificate/private key being basically identical to what provides. Why is this needed: One use case is where the intent is to add HTTPS functionality to lifecycle hooks (postStart/preStop). There is currently no easy way to test this logic because the container which receives the HTTPS request (in this case, ) does not support HTTPS traffic.
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing


Question: utilruntime.HandleError(fmt.Errorf("Unable to get pod %v/%v's service memberships: %v", oldPod.Namespace, oldPod.Name, err)) return } <del> services = determineNeededServiceUpdates(oldServices, services, podChanged) </del> <ins> services = determineNeededServiceUpdates(oldServices, services, podChangedFlag) </ins> } for key := range services {

This test has started failing on our e2e test suite, erroring out with "Failed to get endpoints..." https://k8s- https://k8s- Example failures https://k8s- https://k8s- cc cc
This may be related to our recently-merged endpoints change in . I'll look into it.
I verified that the flakiness introduced was a result of . A fix in should be merging soon. /assign
GitHub didn't allow me to assign the following users: joelsmith. Note that only can be assigned. <detailsIn response to : I verified that the flakiness introduced was a result of . A fix in should be merging soon. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: fs.DurationVar(&o.config.IPVS.SyncPeriod.Duration, "ipvs-sync-period", o.config.IPVS.SyncPeriod.Duration, "The maximum interval of how often ipvs rules are refreshed (e.g. '5s', '1m', '2h22m').  Must be greater than 0.") fs.DurationVar(&o.config.IPVS.MinSyncPeriod.Duration, "ipvs-min-sync-period", o.config.IPVS.MinSyncPeriod.Duration, "The minimum interval of how often the ipvs rules can be refreshed as endpoints and services change (e.g. '5s', '1m', '2h22m').") fs.StringSliceVar(&o.config.IPVS.ExcludeCIDRs, "ipvs-exclude-cidrs", o.config.IPVS.ExcludeCIDRs, "A comma-separated list of CIDR's which the ipvs proxier should not touch when cleaning up IPVS rules.") <ins> fs.BoolVar(&o.config.IPVS.StrictARP, "ipvs-strict-arp", o.config.IPVS.StrictARP, "Enable strict ARP by setting arp_ignore to 1 and arp_announce to 2") </ins> fs.DurationVar(&o.config.ConfigSyncPeriod.Duration, "config-sync-period", o.config.ConfigSyncPeriod.Duration, "How often configuration from the apiserver is refreshed.  Must be greater than 0.") fs.BoolVar(&o.config.IPTables.MasqueradeAll, "masquerade-all", o.config.IPTables.MasqueradeAll, "If using the pure iptables proxy, SNAT all traffic sent via Service cluster IPs (this not commonly needed)") fs.StringVar(&o.config.ClusterCIDR, "cluster-cidr", o.config.ClusterCIDR, "The CIDR range of pods in the cluster. When configured, traffic sent to a Service cluster IP from outside this range will be masqueraded and traffic sent from pods to an external LoadBalancer IP will be directed to the respective cluster IP instead")

/kind bug Kubernetes version: 1.13.1. kube-proxy mode: IPVS. SDN: flannel. We are used services like this: where IP 192.168.10.201 is from our internal /24 network (in particular, our Kube nodes have IPs 192.168.10.1, 192.168.10.2 etc). Note that there is no conflicts between real IPs and virtual external ones. Before updating from 1.12.3 to 1.13.1, it worked correctly. In partucular these IPs was accessible from other non-Kubernetes hosts from 192.168.10.0/24 network. After update kube-proxy to 1.13.1, this feature stops working: non-Kubernetes hosts cannot resolve service IPs via ARP: On Kubernetes hosts it still working, because not requires ARP resolving. If we reset sysctl parameter arpignore on Kube hosts, all works normally even from non-Kube hosts. Quck debugging shows that problem is in the commit . Please make this behaviour opt-out. (Now we resets arpignore to 0 via cron every 1 min.)
/sig network
Having the same problem on Kubernetes 1.13.2 btw Thanks for a workaround!
Same problem, fix this please!!!
This is similar to We've had a discussion there on how to best implement a solution but haven't decided yet I'm in favor of flag which defaults to the previous behavior to avoid breaking existing setups (alternatives are cleaner but involve creating an additional arptable rule per service which I think we should avoid) what do you think?
OK

What happened: After PR: , we noticed communications issue in our pods. We use the Lyft CNI plugin () which relies on two interfaces in pod: an IPVLAN interface connecting to the AWS VPC which has a VPC IP an Unnumbered point-to-point interface (veth) connecting to the host and that does not have an IP (it "borrows" the one from the IPVLAN interface) On the host side, there is a veth interface which also doesn't have an IP (it borrows one from the main interface). With the latest versions of kube-proxy communication between host and pods break What you expected to happen: Communication still works How to reproduce it (as minimally and precisely as possible): Use the Lyft CNI plugin Deploy kube-proxy Try to ping the host from a pod (or a pod from the host) Anything else we need to know?: The new sysctls create 2 issues: containers started after kube-proxy inherit so they will not answer ARP queries on the veth interface the veth on the host side also does not have an IP so it will no answer ARP queries from the pods I know that our setup is pretty uncommon but assuming that the veth on the host side will have an IP seems dangerous. Some discussions on the PTP plugin for instance suggest not using IP for the host interface: containernetworking/plugins Possible fixes: create a flag to enable ARP filtering (I think to avoid breaking existing setups, we should disable it by default but I'm fine with enabling it by default as long as we document the solution) use arptables to filter ARP queries for clusterIPs (): a bit more work because we need to do it for all clusterIPs (there may be a more efficient way to do it but I could not make it work by filtering on the kube-ipvs interface) and this requires the arptables binary. We could test for the existence of the binary and log a warning if it is missing (similar to behavior with conntrack) Environment: Kubernetes version (use ): kube-proxy OS (e.g. from /etc/os-release): Kernel (e.g. ): /area ipvs /sig network /kind bug /assign ping
I can say only that solution with arptables working fine with MetalLB.
Oh, really? Glad you confirm that.

But I would like to make sure that we won't create too many arptables rules. Can ipset works with arptables?
It doesn't look like arptables works with ipset
Unfortunately doesn't support ipsets, but new do. (not fully implemented yet) Eg we can create ipset for arp: But current implementation is not allowing to use it for arp rules for now: Will return: I guess because there is still no type specified for (destination ip) and (source ip) BTW: nft does not allow to create set of integer components: But there is a way for specify multiple IPs in one rule, eg: Where is IPv4 addresses in decimal format More info here:
Another option would be to use the and filter all queries but this flag is not in kube-proxy today
Bad idea, it will handle but not - which can be really differ
can you explain more detail why Lyft CNI can not assign IPs to the right interfaces for avoid this kind of issues?
Yes you are right. bad idea It could definitely add IPs to these interfaces but it's not the choice that was made. More details here: ("Unnumbered point-to-point interface"). I could probably PR a fix there. I'm a bit worried that some other CNI plugins could break here though
We are about to deploy some 1.13 clusters and can't use kube-proxy 1.13 as is We could start with a custom build but I assume others will be impacted The simplest way forward for now is probably to use a flag to control this behavior. I think the safest is probably to disable this by default to avoid the issue for people upgrading but if you think it is really important we can enable it by default. I know we'd like to avoid adding more flags but other solutions seem a bit more complicated and will most likely result in one iptables rule per service. What do you think ?
Just FYI, we have another CNI plugin that is affected by this.
We addressed the issue by assigning a link-local address 169.254.1.1/32 to the host-side veth in . So this issue is not critical at least for our case.
Good news everyone, new nft supports setting ip address for arp filters : that's makes possible for using sets


Question: ) func main() { <ins> klog.InitFlags(nil) </ins> flag.Usage = func() { fmt.Fprintf(os.Stderr, "Usage: %s [FLAG]... [PROFILE_DIR]...n", os.Args[0]) fmt.Fprintf(os.Stderr, "Load the AppArmor profiles specified in the PROFILE_DIR directories.n")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: docker image built from source doesn't work with below error message, run binary doesn't work either, What you expected to happen: profile could be loaded successfully. How to reproduce it (as minimally and precisely as possible): cd test/images make all WHAT=apparmor-loader create namespace create DaemonSet with the built image check the log of the pod, kubectl logs apparmor-loader-dcbvp -n apparmor Anything else we need to know?: should init the flags in the source. Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ): all Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: ```console $ kubectl run nginx --image=nginx --replicas=1 --namespace=limit-example <del> CONTROLLER   CONTAINER(S)   IMAGE(S)   SELECTOR    REPLICAS nginx        nginx          nginx      run=nginx   1 </del> <ins> replicationcontroller "nginx" created </ins> $ kubectl get pods --namespace=limit-example <del> POD           IP           CONTAINER(S)   IMAGE(S)   HOST          LABELS      STATUS    CREATED          MESSAGE nginx-ykj4j   10.246.1.3                             10.245.1.3/   run=nginx   Running   About a minute nginx          nginx                                Running   54 seconds $ kubectl get pods nginx-ykj4j --namespace=limit-example -o yaml | grep resources -C 5 </del> <ins> NAME          READY     STATUS    RESTARTS   AGE nginx-aq0mf   1/1       Running   0          35s $ kubectl get pods nginx-aq0mf --namespace=limit-example -o yaml | grep resources -C 8 </ins> ``` ```yaml <ins> resourceVersion: "127" selfLink: /api/v1/namespaces/limit-example/pods/nginx-aq0mf uid: 51be42a7-7156-11e5-9921-286ed488f785 spec: </ins> containers: <del> - capabilities: {} image: nginx </del> <ins> - image: nginx </ins> imagePullPolicy: IfNotPresent name: nginx resources: limits: <del> cpu: 250m </del> <ins> cpu: 300m memory: 200Mi requests: cpu: 200m </ins> memory: 100Mi terminationMessagePath: /dev/termination-log volumeMounts: ``` <del> Note that our nginx container has picked up the namespace default cpu and memory resource limits. </del> <ins> Note that our nginx container has picked up the namespace default cpu and memory resource *limits* and *requests*. </ins> Let's create a pod that exceeds our allowed limits by having it have a container that requests 3 cpu cores. ```console $ kubectl create -f docs/admin/limitrange/invalid-pod.yaml --namespace=limit-example <del> Error from server: Pod "invalid-pod" is forbidden: Maximum CPU usage per pod is 2, but requested 3 </del> <ins> Error from server: error when creating "docs/admin/limitrange/invalid-pod.yaml": Pod "invalid-pod" is forbidden: [Maximum cpu usage per Pod is 2, but limit is 3., Maximum cpu usage per Container is 2, but limit is 3.] </ins> ``` Let's create a pod that falls within the allowed limit boundaries. ```console $ kubectl create -f docs/admin/limitrange/valid-pod.yaml --namespace=limit-example <del> pods/valid-pod $ kubectl get pods valid-pod --namespace=limit-example -o yaml | grep -C 5 resources </del> <ins> pod "valid-pod" created $ kubectl get pods valid-pod --namespace=limit-example -o yaml | grep -C 6 resources </ins> ``` ```yaml <ins> uid: 162a12aa-7157-11e5-9921-286ed488f785 spec: </ins> containers: <del> - capabilities: {} image: gcr.io/google_containers/serve_hostname </del> <ins> - image: gcr.io/google_containers/serve_hostname </ins> imagePullPolicy: IfNotPresent <del> name: nginx </del> <ins> name: kubernetes-serve-hostname </ins> resources: limits: cpu: "1" memory: 512Mi <del> securityContext: capabilities: {} </del> <ins> requests: cpu: "1" memory: 512Mi </ins> ``` <del> Note that this pod specifies explicit resource limits so it did not pick up the namespace default values. </del> <ins> Note that this pod specifies explicit resource *limits* and *requests* so it did not pick up the namespace default values. Note: The *limits* for CPU resource are not enforced in the default Kubernetes setup on the physical node that runs the container unless the administrator deploys the kubelet with the folllowing flag: ``` $ kubelet --help Usage of kubelet .... --cpu-cfs-quota[=false]: Enable CPU CFS quota enforcement for containers that specify CPU limits $ kubelet --cpu-cfs-quota=true ... ``` </ins> Step 4: Cleanup ----------------------------

Since resource QoS is implemented, limitrange also supports request constraint, we can add an example for that.
cc


Question: return scale, foundSpecReplicas, nil } <ins> type scaleUpdatedObjectInfo struct { reqObjInfo         rest.UpdatedObjectInfo specReplicasPath   string statusReplicasPath string labelSelectorPath  string } func (i *scaleUpdatedObjectInfo) Preconditions() *metav1.Preconditions { return i.reqObjInfo.Preconditions() } func (i *scaleUpdatedObjectInfo) UpdatedObject(ctx context.Context, oldObj runtime.Object) (runtime.Object, error) { cr := oldObj.DeepCopyObject().(*unstructured.Unstructured) const invalidSpecReplicas = -2147483648 // smallest int32 oldScale, replicasFound, err := scaleFromCustomResource(cr, i.specReplicasPath, i.statusReplicasPath, i.labelSelectorPath) if err != nil { return nil, err } if !replicasFound { oldScale.Spec.Replicas = invalidSpecReplicas // signal that this was not set before } obj, err := i.reqObjInfo.UpdatedObject(ctx, oldScale) if err != nil { return nil, err } if obj == nil { return nil, apierrors.NewBadRequest(fmt.Sprintf("nil update passed to Scale")) } scale, ok := obj.(*autoscalingv1.Scale) if !ok { return nil, apierrors.NewBadRequest(fmt.Sprintf("wrong object passed to Scale update: %v", obj)) } if scale.Spec.Replicas == invalidSpecReplicas { return nil, apierrors.NewBadRequest(fmt.Sprintf("the spec replicas field %q cannot be empty", i.specReplicasPath)) } specReplicasPath := strings.TrimPrefix(i.specReplicasPath, ".") // ignore leading period if err := unstructured.SetNestedField(cr.Object, int64(scale.Spec.Replicas), strings.Split(specReplicasPath, ".")...); err != nil { return nil, err } if len(scale.ResourceVersion) != 0 { // The client provided a resourceVersion precondition. // Set that precondition and return any conflict errors to the client. cr.SetResourceVersion(scale.ResourceVersion) } return cr, nil } </ins>

What happened: Using a command such as: fails with the following error: This bug happens on each run and started occurring as of v1.15.0. With v1.14 or older, I don't have such problem, i.e. it works flawlessly. What you expected to happen: The command succeeds: How to reproduce it (as minimally and precisely as possible): Create a CRD that has the scale subresource and try to scale a CR using . Example of such CRD can be found . Anything else we need to know?: Regarding the bug, it seems that it started appearing as of . Before the PR, we always fetched the resource and it worked because it had the resource version, while now we are only fetching it in case preconditions are provided (preconditions are not provided by default/when using ). Here is the output of a command: If I manually craft the request that has the resource version (or use the flag), it works. Is it as expected that the Scale resource requires the resource version in order for update to succeed? Shouldn't it always pick the latest resource version in case it isn't provided? On that note, I also noticed that if you want to update a CR by crafting the request manually and using , you also need to specify the resource version or you will get the same error. I'm not sure if this is expected and related, but I'm writing it down as it might be related. As far as I know, this is not required for some native resources at all, but I'm not sure is this the case for CRDs. Environment: Kubernetes version (use ):
/sig api-machinery cc
/sub
could you try to see if it fix the problem?
I've created a test cluster based on and I can confirm that I'm able to scale CustomResources using as usual. I used an to test this, such as: Thanks a lot for your time spent fixing this!
Bug triage for 1.17 here with a gentle reminder that code freeze for this release is on November 14. Is this issue still intended for 1.17? I see you move it to 1.17 so I just wanted to be sure.
I guess could be merged now?

It is in the review queue and should still make 1.17
a kind reminder that code freeze for this release is on November 14.
I wanted to circle back around on this. We are 4 days out from code freeze. If this is still good based on just note here. Judging from the looks of it we are still good
Yes this should merge early in the week
Code freeze is tomorrow, this still going to make it?
yes, is still planned


Question: } }) <del> ginkgo.It("should validate PreferredVersion for each APIGroup", func() { </del> <ins> /* Release : v1.19 Testname: Discovery, confirm the PreferredVersion for each api group Description: Ensure that a list of apis is retrieved. Each api group found MUST return a valid PreferredVersion unless the group suffix is example.com. */ framework.ConformanceIt("should validate PreferredVersion for each APIGroup", func() { </ins> // get list of APIGroup endpoints list := &metav1.APIGroupList{}

] 20 July 2020 [x] Two weeks soak end date: 3 August 2020 According to this APIsnoop query, there are still an endpoint which is untested. Get a list of all group /apis Iterate through list Query each endpoint List the PreferredVersion for the endpoint Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 15 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/retitle Write checkAPIGroupPreferredVersion Test +Promote - +16 Endpoint coverage
/reopen Merged /assign Ready for creation of Promotion PR
Reopened this issue. <detailsIn response to : /reopen Merged /assign Ready for creation of Promotion PR Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return 1 fi <ins> if [[ ${GCE_PRIVATE_CLUSTER:-} == "true" ]]; then if gcloud compute routers describe --project "${NETWORK_PROJECT}" --region "${REGION}" "${NETWORK}-nat-router" &>/dev/null; then KUBE_RESOURCE_FOUND="Cloud NAT router" return 1 fi fi </ins> # No resources found. return 0 }

Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Create a iptables utils. execer := exec.New() <del> iptInterface = utiliptables.New(execer, protocol) </del> kernelHandler = ipvs.NewLinuxKernelHandler() ipsetInterface = utilipset.New(execer) canUseIPVS, err := ipvs.CanUseIPVSProxier(kernelHandler, ipsetInterface)

What happened: Currently kube-proxy cleanup not work, which will left iptables (and ipvs) rules in host and. What you expected to happen: Kube-proxy cleanup can work and not relay on hostName or ClientConnection config. How to reproduce it (as minimally and precisely as possible):
/sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


Question: containerRefManager, kubeDeps.Recorder) <del> tokenManager := token.NewManager(kubeDeps.KubeClient.CoreV1()) </del> <ins> tokenManager := token.NewManager(kubeDeps.KubeClient) </ins> klet.volumePluginMgr, err = NewInitializedVolumePluginMgr(klet, secretManager, configMapManager, tokenManager, kubeDeps.VolumePlugins, kubeDeps.DynamicPluginProber)

/kind bug /priority failing-test /priority important-critical /milestone v1.11 /assign cc All GKE jobs are failing in various release dashboards. Code freeze for 1.11 is a couple of hours away.
Common error appears to be:
Failures seem to align with when was merged. But nothing in that PR seems suspicious. Having verify. Bringing up my own cluster on head to try and get a live cluster.
I have a quick try to reproduce one of the failure test cases by gcloud container clusters create --addons=HttpLoadBalancing,HorizontalPodAutoscaling,KubernetesDashboard --machine-type=n1-standard-2 --image-type=gci --num-nodes=3 --cluster-version=1.11.0-beta.1.222+ test- The cluster isn't up. What I observed is the following: 1) kubelet on master is in crash-loop due to a panic No other master components could be up. 2) kubelet is in crashloop in node as well.
/sig node Looks like has a patch up at
/priority critical-urgent
/status approved-for-milestone
Waiting for new runs to verify the fix. Will reopen this issue if needed
/reopen Reopening this one until its actually verified to ensure it shows up in our tracking queries.
I see the GKE tests cleaning up in the . still doesn't seem to have the change (will have to wait for cutover?). The fix looks promising. Will wait for a couple more runs before I close
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Action Required: This issue has not been updated since Jun 6. Please provide an update. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release.

</details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
all the GKE ones still look good /close


Question: The replication controller is forever constrained to this narrow responsibility. It itself will not perform readiness nor liveness probes. Rather than performing auto-scaling, it is intended to be controlled by an external auto-scaler (as discussed in [#492](https://github.com/GoogleCloudPlatform/kubernetes/issues/492)), which would change its `replicas` field. We will not add scheduling policies (e.g., [spreading](https://github.com/GoogleCloudPlatform/kubernetes/issues/367#issuecomment-48428019)) to replication controller. Nor should it verify that the pods controlled match the currently specified template, as that would obstruct auto-sizing and other automated processes. Similarly, completion deadlines, ordering dependencies, configuration expansion, and other features belong elsehwere. We even plan to factor out the mechanism for bulk pod creation ([#170](https://github.com/GoogleCloudPlatform/kubernetes/issues/170)). <del> The replication controller is intended to be a composable building-block primitive. We expect higher-level APIs and/or tools to be built on top of it and other complementary primitives for user convenience in the future. The "macro" operations currently supported by kubecfg (run, stop, resize, rollingupdate) are proof-of-concept examples of this. For instance, we could imagine something like [Asgard](http://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html) managing replication controllers, auto-scalers, services, scheduling policies, canaries, etc. </del> <ins> The replication controller is intended to be a composable building-block primitive. We expect higher-level APIs and/or tools to be built on top of it and other complementary primitives for user convenience in the future. The "macro" operations currently supported by kubectl (run-container, stop, resize, rollingupdate) are proof-of-concept examples of this. For instance, we could imagine something like [Asgard](http://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html) managing replication controllers, auto-scalers, services, scheduling policies, canaries, etc. </ins> ## Common usage patterns

It's causing problems and is kind-of GCE dependent anyway, we should just use the kubecfg binary, and document the environment variables it uses.
I think its beneficial to keep something, but remove the GCE dependent pieces, i.e. "external-ip" piece.
I put together some instructions here:
We can remove this when kubecfg is deprecated .
Reopened since it still needs to be removed. /cc
Doh. Didn't mean to declare this fixed in that PR.
Closing this (again) as it is a dupe of


Question: func getStatsSupportedSubsystems(cgroupPaths map[string]string) (*libcontainercgroups.Stats, error) { stats := libcontainercgroups.NewStats() <del> for _, sys := range getSupportedSubsystems() { </del> <ins> for sys, required := range getSupportedSubsystems() { </ins> if _, ok := cgroupPaths[sys.Name()]; !ok { <del> return nil, fmt.Errorf("Failed to find subsystem mount for subsystem: %v", sys.Name()) </del> <ins> if required { return nil, fmt.Errorf("Failed to find subsystem mount for required subsystem: %v", sys.Name()) } // the cgroup is not mounted, but its not required so continue... glog.V(6).Infof("Unable to find subsystem mount for optional subsystem: %v", sys.Name()) continue </ins> } if err := sys.GetStats(cgroupPaths[sys.Name()], stats); err != nil { return nil, fmt.Errorf("Failed to get stats for supported subsystems : %v", err)

If I run first time it goes fine (after reboot). For subsequent runs I get nodes NotReady. I've tried restarting docker but that won't help. Before christams this used to work all the time. Running Arch Linux if it helps. In I see
/sig node
ditto for me: Looking for a work around I noticed that HugePages recently moved to beta. I can get pods to start again by disabling the HugePages feature gate .
/assign
cc
default Arch kernel doesn't have a hugetlb cgroup controller compiled in(been there done that, I use Arch as well), so you have to prepare your own kernel to run it.
If it were a kernel problem I would have expected the kubelet to fail the first time it was started rather than the second time. This isn't confined to Arch, though. I'm having the same problem with the default Debian kernel:
can you check if your Kernel was compiled with flag? You can do it by running: I bet we have to add documentation, describing this behavior. On Ubuntu 16.04 machine:
Sure, looks like it was not: Documentation (especially in the error message) would be great, but if the kubelet isn't going to survive a restart, I would prefer that it not start the first time. Better to fail immediately than eventually.
can you please check if works (as in fails!) for you?
No, this doesn't resolve the issue. Logs from the first start: Logs from the second start:
Do you have the directory? in that case what i have is insufficient :( If you don't have that directory, then we should probably look for a better spot to put that check? any advice?
No, that directory is missing on my system:
I'm seeing this issue as well.
I am preparing a fix to not fail when hugetlb mount is not present. Will post a fix shortly.
I posted a fix here:


Question: Ports:     []v1.EndpointPort{}, }}, }) <ins> if !reflect.DeepEqual(originalService, service) { t.Fatalf("syncing endpoints changed service: %s", diff.ObjectReflectDiff(service, originalService)) } </ins> endpointsHandler.ValidateRequestCount(t, 1) endpointsHandler.ValidateRequest(t, testapi.Default.ResourcePath("endpoints", ns, "foo"), "PUT", &data) }

bug (): Originally posted by in cc /sig network /priority critical-urgent
fix in
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: if template.Spec.RestartPolicy != api.RestartPolicyAlways { allErrs = append(allErrs, field.NotSupported(fldPath.Child("spec", "restartPolicy"), template.Spec.RestartPolicy, []string{string(api.RestartPolicyAlways)})) } <ins> if template.Spec.ActiveDeadlineSeconds != nil { allErrs = append(allErrs, field.Invalid(fldPath.Child("spec", "activeDeadlineSeconds"), template.Spec.ActiveDeadlineSeconds, "must not be specified")) } </ins> } return allErrs }

setting activeDeadlineSeconds in deployment pod template, and seen new pods are continuously created after the older one dies. Is this excepted? if not we should forbidden this.
Yes, it is expected. Once a pod dies, the replica set controller will spawn a new one. activeDeadlineSeconds doesn't make any sense when used under a controller.
we may want to make it a validation error, but I don't feel strong.
Yes, we should not allow activeDeadlineSeconds for restartPolicy=Always pods generally.
What if I want a container to be restarted no matter what status it exits with, but to give up trying if it cannot even start up in seconds? So like this: Start a pod if it keeps constantly failing for seconds: stop it else (if it managed to start and not fail in those first seconds from creating): restart it everytime it fails or successfully finishes
then you want your own controller, RCs are designed for a simple use case: long-running apps. Have a look in , though keep in mind they are under a major redesign:
Sorta late to the party, but this behavior stops us from being able to use a Deployment to mitigate a production memory leak. Our application can handle a , but a leaves items in a pending state for our customers. Killing pods on a regular basis through kubernetes would allow us to induce known behavior of a graceful termination versus deploying new code to handle a timer before terminating. While I understand the intention of this feature, I feel that as a mitigation tool while my teams track down the memory leak, this would let my oncall rotation sleep easy.


Question: } func (rl *respLogger) AddKeyValue(key string, value interface{}) { <ins> rl.mutex.Lock() defer rl.mutex.Unlock() </ins> rl.addedKeyValuePairs = append(rl.addedKeyValuePairs, key, value) }

Apiserver crashed with stacktrace: Not to crash I was able to reproduce it by "slowing down" deleting headers in Authentication filter (): And making requests (around 5-10 requests to trigger this race condition): Crash was caused by race condition - one goroutine was modifying headers of requests (for example authentication filter). The other goroutine was trying to get UserAgent from request, which is also taken from request's headers map. This causes unrecoverable panic. <detailsAll versions, tested on 1.21 </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery /priority important-soon
/triage accepted
Reopening - since we need to cherrypick it to previous releasses.
- friendly ping on opening cherrypicks
I've opened two cherry-picks:
One more for 1.20:
All merged - thanks!


Question: Type=oneshot RemainAfterExit=yes ExecStartPre=/usr/bin/mkdir -p /opt/cni <del> ExecStartPre=/usr/bin/curl --fail --silent --show-error --location --create-dirs --output /opt/downloads/cni.tar.gz https://storage.googleapis.com/kubernetes-release/network-plugins/cni-9d5e6e60e79491207834ae8439e80c943db65a69.tar.gz </del> <ins> ExecStartPre=/usr/bin/curl --fail --silent --show-error --location --create-dirs --output /opt/downloads/cni.tar.gz https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz </ins> ExecStart=/usr/bin/tar xf /opt/downloads/cni.tar.gz -C /opt/cni/ - name: kubernetes-install-docker2aci.service

I have a container that got the broadcast address of podCIDR subnet. My CNI conf is: And the container in question has which is the broadcast address for the subnet. So it can't figure out the MAC of cbr0 because all frames are getting sent to (or at least, that's what seems to indicate). Eg: tcpdump alonside a wget on vs a similar normal container
Looks like a bug in ipam. Will try to put together a fix.
I stress tested your fix overnight with the networking suite I'm writing (the one that found the bug), didn't hit the failure in 100 runs, we should get it in asap
Great. It would be better to pull from upstream once the fix is merged. Otherwise, we have to fork it like we did in 1.3.
Marking this p0 flake so it shows up in release queries
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: // inherited features from apiextensions-apiserver, relisted here to get a conflict if it is changed // unintentionally on either side: apiextensionsfeatures.CustomResourceValidation: {Default: true, PreRelease: utilfeature.Beta}, <del> SupportIPVSProxyMode:                           {Default: false, PreRelease: utilfeature.Alpha}, </del> }

There are some work to be done in the follow-up of , that is, help IPVS existing alpha. [x] GCE CI failures()[x] cross-node visit NodePort ()[x] hairpin issue ()[x] Using ipset doing snat and packet filtering and reduce iptables call().[x] automatically load kernel modules. (, )[x] creating e2e test suite for IPVS in upstream CI ().[x] ip cmd -netlink bind/unbind IP address to/from dummy device()[x] rsync to HEAD of iptables(, )[x] enable flush batch API()[x] Fix TODOs in codes ()[x] Bug report (, )[x] Default sync period()[x] IPV6 support(, and more...) /cc
/cc
ref ! It will generate annoy error info If unbind multi-ports on one service .
You're right. I think it's a nice finding. Can you fire a PR to fix it?
Sure, I'll make it soon.
I found it is treated as already bound if returned 2 in , but ref to the and of ip , it just means an error was reported by the kernel, such as "Operation not permitted". I prefer to use instead of command to manage ip address.
Actually, I have considered to use netlink before. I think what you said make sense, but can we discuss it in other thread? I mean, create a new issue and link it here? Thanks!
I opened a issue
on the subject :)
OOPS... I will fix that typo. Thanks
Wish to use ipset to replace KUBE-MARK-MASQ chain.
I find it's a reasonable suggestion and am trying to make it workable...
There are to ways to add IPVS CI job: , but doesn't support any kube-proxy config right now, so I had this pr , and also, need support it too, see this pr: kubernetes/kubernetes-anywhere build cluster. See this pr: kubernetes/test-infra The second way is much easier and faster, but I was told they gonna deprecated the , so in the long term, we still need and 's support. If anybody have better way or have some suggestion about this, please let me know, I'll be really appreciate, Thanks.
This one is done as both PRs are merged:
ACK!

/area ipvs
We are almost reaching the beta, except GCE CI failures, see The failure is the case that, visiting Cluster IP from Pod A, and the backend Pod B is in the same host with the Pod A. Pod A failed to get response from Pod B until timeout. I can reproduce the failures in my GCE cluster with kubent as network plugin, but it works fine in my local-up cluster and people has never reported the issue yet. and I are digging...
IPVS CI is green now.


Question: "golang.org/x/net/websocket" v1 "k8s.io/api/core/v1" <ins> apierrors "k8s.io/apimachinery/pkg/api/errors" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime"

pull-kubernetes-conformance-kind-ga-only-parallel [sig-node] Pods should run through the lifecycle of Pods and PodStatus [Conformance] After is merged. It relies on events which are not reliable as discussed . No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The test is not failing that much looking at the testgrid. Is it actually an issue?
Not so often, but it does happen. So, I think that it is worth fixing.
still flaking -
flaking conformance tests are concerning and should take priority over new conformance test additions
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale I still see this test failing , also in periodic jobs
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale seems like it's is still failing
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
still flake /remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/assign Namanl2001
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
please unassign if you are not working on it. Seems to be still flaking
It still flakes but not that much. I checked the history: adjusted the timeout from 30s to 1m. changed it not to fail for watching Delete Event, and just print it like below: This time, we see the above adding warning and then failed. Looking into kubelet/container runtime log. The pod is just deleted after 1 min, maybe 1min10s. I'm afraid that we have to adjust the timeout of the delete pod again to fix it. Is a valid pod deletion timeout?

I think most deletion timeout of pod is 5min, however this is too long after pod with .
but why it takes so long kubelet to delete a pod? what are all of those ""Failed to delete cgroup paths" messages? can this be hiding some problem with runc or the kubelet?
Sep 19 13:14:56 kinder-external-etcd-worker-2 kubelet[359]: E0919 13:14:56. 359 ] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods besteffort pod53d19839-496a-4be8-8964-] : unable to destroy cgroup paths for cgroup [kubepods besteffort pod53d19839-496a-4be8-8964-] : Timed out while waiting for systemd to remove kubepods-besteffort-" pod="pods-900/pod-test" podUID=-496a-4be8-8964- Sep 19 13:14:56 kinder-external-etcd-worker-2 kubelet[359]: I0919 13:14:56. 359 ] "Failed to delete cgroup paths" cgroupName=[kubepods besteffort pod53d19839-496a-4be8-8964-] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod53d19839-496a-4be8-8964-] : Timed out while waiting for systemd to remove kubepods-besteffort-" Sep 19 13:14:56 kinder-external-etcd-worker-2 containerd[220]: time="2023-09-19T13:14:56.925525892Z" level=info msg="StartContainer for "e3b435116d2640d7c8166332e1c4bf6722f97ac1401035cbe443704c8b442efd" returns successfully" Sep 19 13:14:56 kinder-external-etcd-worker-2 kubelet[359]: I0919 13:14:56.

359 ] "Failed to delete cgroup paths" cgroupName=[kubepods besteffort pod53d19839-496a-4be8-8964-] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod53d19839-496a-4be8-8964-] : Timed out while waiting for systemd to remove kubepods-besteffort-" Sep 19 13:15:01 kinder-external-etcd-worker-2 kubelet[359]: I0919 13:15:01. 359 ] "Cleaned up orphaned pod volumes dir" podUID=-496a-4be8-8964- path="/var/lib/kubelet/pods/-496a-4be8-8964-/volumes" It means timeout. Another failure is not for the same reason.
/joke
I wish I could clean mirrors for a living. It's just something I can see myself doing. <detailsIn response to : /joke Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is starting to be worrisome
another failure
was another occurrence of this I think. I'm seeing the timeout error for deleting the pod in the build logs: And then the error: It has the same sanbok not found error in the kind-worker logs: Along with the same cgroup error in the logs: So it's probably an instance of the first case mentioned caused by timeout.
Flake in . Delete timeout is 1 minute here. It still need more investigation why this is not deleted in 1 min.(heavy load or logic problems.)
Flake twice in yersterday.


Question: - 'go': branch: 'master' cron-string: '{sq-cron-string}' <del> timeout: 80 </del> <ins> timeout: 100 </ins> - 'go-release-1.2': branch: 'release-1.2' # Every 3 hours cron-string: 'H H/3 * * *' <del> timeout: 50 </del> <ins> timeout: 100 </ins> - 'go-release-1.1': branch: 'release-1.1' # Every 6 hours cron-string: 'H H/12 * * *' <del> timeout: 30 </del> <ins> timeout: 60 </ins> jobs: - 'kubernetes-test-{suffix}'

Happened twice in a row.
Can you take a look? I think the gke build has the same problem.
Looks like they've gotten significantly slower since the last time this came up. I'll try and figure out why.
This happened again and is continuing to block the merge queue. can you make a bandaid that extends the timeout?
Done, but I need to stress that we need to know why the duration is climbing. I'm afraid in a few months I'll get another issue that says "kubernetes-test-go is timing out".
Thanks. Yes, we can leave this issue open until we figure it out.
maybe another bandaid for our build?
Actually, it was taking 14-17 mins, so 30 minute timeout should hopefully be enough. hm.
The build is the result of a PR going in that does affect the build. It had to download a new build image, and it also needs to build more tarballs. We might need to legitimately bump that one's timeout.
It looks like it's about to time out again.
Barely passed. The build got slower by a factor of two after went in. This was somewhat expected. I'll bump the timeout.
We should exclude those platforms in our testing, maybe? We don't need ppc or arm binaries. Should we roll back ?
I don't think we should roll it back. We want to do a full release build, which now includes building for those architectures. However, if the PR Jenkins e2e job starts timing out, then we should probably change it to do a quick-release instead of the full release. cc
Dropped the priority now that the bleeding has stopped. I'll try to figure out why the test-go job got slower.
this is a fairly significant increase in build and release times with . Is there anything we can do to mitigate the increase? I see many packages downloaded during the build. Are we maybe downloading more than we need to? Can we cache anything somehow/somewhere? Can we parallelize package updating and/or building?
The problem is not all the things that get downloaded into , that's just one time. It's the building time that's a small problem. Since building may get ~2x slower, and now we've both upgraded to and more server platforms, so the increase in build time is expected.

Sorry for not notifying you in beforehand though. Steps we could take to decrease the time: Move to test targets. I don't know why that one is considered a server target. Build test targets only for (now , , , ) Remove addon images: Consider dropping support for for Remove for , and . I don't think it's required from official builds.
We should do a full build on every CI run for all arches, so we may detect regressions in the code
Whoops, I meant to make a PR to do that a while ago. cc
thanks for letting me know. Do you want me to send the PR or will you?
I think we definitely want to build test targets for at least darwin, since plenty of people develop on a mac. It might be worth dropping tests for arm and windows. sgtm
All of our kubernetes-build and kubernetes-test-go jobs are running on a single n1-highmem-32 instance, which is hitting 100% CPU usage fairly often. This is most likely why our test-go times are so inconsistent lately.
Is there anything I may help with? (Not having access to your servers)
I think I can handle the test-go problems. Feel free to work on your other suggestions, I think they're good. Thanks, though :)
The timeout is too aggressive: shows a passing run in 75 minutes () and the average passing run time is in the mid 50m. We need at least a 100m timeout (2x average 50m runtime) rather than 80. Or to put it another way: in order to run this job reliably I want the timeout to be twice the average runtime, rather than a couple additional minutes.
I'm wary of a 100 minute timeout. Last time this came up the job was taking ~35 minutes () so an 80 minute timeout was fine. Some time last week it started taking 60-80 minutes and occasionally timing out. I'd like to know why. In the meantime, to fix the submit queue lets bump the timeout. I'm also starting to think we should run verify-*.sh, unit/integration tests, and test- in separate Jenkins jobs. kubernetes-test-go blocks the queue for over an hour now.


Question: return true } <del> func (ca *cloudCIDRAllocator) canRetry(nodeName string) bool { </del> <ins> func (ca *cloudCIDRAllocator) retryParams(nodeName string) (bool, time.Duration) { </ins> ca.lock.Lock() defer ca.lock.Unlock() <del> count := ca.nodesInProcessing[nodeName].retries + 1 </del> <ins> entry, ok := ca.nodesInProcessing[nodeName] if !ok { glog.Errorf("Cannot get retryParams for %q as entry does not exist", nodeName) return false, 0 } count := entry.retries + 1 </ins> if count > updateMaxRetries { <del> return false </del> <ins> return false, 0 </ins> } ca.nodesInProcessing[nodeName].retries = count <del> return true </del> <ins> return true, nodeUpdateRetryTimeout(count) } func nodeUpdateRetryTimeout(count int) time.Duration { timeout := updateRetryTimeout for i := 0; i < count && timeout < maxUpdateRetryTimeout; i++ { timeout *= 2 } if timeout > maxUpdateRetryTimeout { return maxUpdateRetryTimeout } return timeout </ins> } func (ca *cloudCIDRAllocator) removeNodeFromProcessing(nodeName string) {

/kind bug /sig network What happened: Networking tight loop causing GCP Cloud rate limiting to kick in. Causing tests to fail. Suspected call stack: Looks like it could be the root cause of these test failures here: https://k8s- These storage tests are hitting rate limit on the cloud api when trying to detach/delete disks. However, we have exponential backoff for all these functions. But the rate limiting from an external source is affecting the tests. See logs snippet: Note: These logs span exactly 1 minute. Try grepping "rateLimit" or "CIDRNotAvailable" -also note that the CIDRNotAvailable error messages are probably not verbose enough as they do not pinpoint what the actual error is, you can only associate them by correlation in timing and then looking at the code. Shorter snippet from above log file to demonstrate: since this is running on hundreds of nodes hitting this API call should involve some sort of exponential backoff as well as jitter. /cc /assign
/assign
/assign
/sig cloud-provider /sig gcp /milestone v1.12 (since is marked as v1.12)
may we have a priority on this since Code Freeze is next week? :)
Quick question, are we using the default API rate limits quota for scalability projects?
I will defer to for priority because he probably has the most context on how serious this is and what it could affect
Which rate limit are you referring to? (I don't remember us changing it any time) Btw - this is the project being used for that job -
I put up a patch for adding exponential retry
Gentle reminder that for this to be considered for 1.12 past Code Freeze, which is next week, it must have a status/critical-urgent label. I see this is being actively worked on


Question: if !supportedOutputTypes.Has(o.Output) { return fmt.Errorf("--output %v is not available", o.Output) } <ins> supportedSortTypes := sets.NewString("", "name", "kind") if len(o.SortBy) > 0 { if !supportedSortTypes.Has(o.SortBy) { return fmt.Errorf("--sort-by accepts only name or kind") } } </ins> return nil }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a sort parameter (or similar concept) to the command. Why is this needed: With the numerous amount of API resources (not counting custom ones), it is sometimes difficult locating them in the output. By default, API resources are not sorted by the name column (the first on the left) and often this is the column holding the information one wishes to query. A sort parameter would help more easily identify the object to be understood.
/sig usability


Question: import ( "context" <ins> "encoding/json" "fmt" </ins> "testing" "time"

Why PatchService method is used for deleting finalizers in cloud-controller-manager, This method updates with service/status, It seems to go unnoticed by the API server, resulting in the API server being unaware when the finalizer field on the service is deleted, leading to a delay in the release of service-related NodePorts. service nodeport be released promptly when service is deleted. create a loadblancer service with nodeport seted,then delete it and recreate. cloud-controller-manager add a finalizer on service. delete this service and then cloud-controller remove finalizer. but nodeport is not released when service deleted. ref i change like this, it seems like releasing nodeport when service is deleted : No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
Are you sure? that method updates the status and remove the finalizer ... otherwise , who removes the finalizer?
I apologize for any confusion. What I meant was that the update of the status and the removal of the finalizer are performed through updating service/status. Therefore, removing the finalizer does not release resources.It seems like removal of the finalizer by service/status, store.AfterDelete is nil. i think remove the finalizer should by patch servcie and updates the status should by patch service/status.Now we patch both by servcie/status.
no need to apologize, let me see if I can reproduce this locally first, Services api allocation is complex , it overrides the standard AfterDelete and release all allocated resources there /assign
/sig network ack the problem
will look for a possibly-related past cloud provider issue.
this will be fixed by , it is a problem in the apiserver logic
Cool! I forgot that you can patch finalizers thru /status. I repro'ed it on both ClusterIP and NodePort.

Saw this in the logs: So the repair loop, yet again, saves our butt but also hides a bug.
we need metrics here about to have visibility of the repairs
I also found this problem in versions 1.23 and 1.27
yep, I started the backports , but we can only backport to the supported versions, 1.23 and 1.24 are EOL


Question: tags = ["automanaged"], visibility = ["//visibility:public"], ) <ins> go_test( name = "go_default_test", srcs = ["types_test.go"], embed = [":go_default_library"], deps = ["//vendor/github.com/google/go-cmp/cmp:go_default_library"], ) </ins>

What would you like to be : Actually disable scheduler plugins according to the disabled list in the scheduler's component config. The semantics are defined here: Part of /sig scheduling /priority important-soon /assign
The following semantics seems vague, and I think we should reach consensus on some cases when disabled plugins are involved. Suppose we have two default plugins, and , the runs before , here are the cases we could meet with the scheduling framework, plugins which run after all the default plugins plugins which should run ahead of the plugins plugin which should run ahead of all the plugins -enable default or custom plugin should return errors
Do you have any other cases not mentioned here?
Those cases sound right, I don't have others.
Is this actually what we want? I don't think so. The comment needs rewording.
I plan to revisit the comments after


Question: } func TestProxyUpdatePort(t *testing.T) { <del> port, err := echoServer(t, "127.0.0.1:0") if err != nil { t.Fatal(err) } </del> lb := NewLoadBalancerRR() lb.OnUpdate([]api.Endpoints{{JSONBase: api.JSONBase{ID: "echo"}, Endpoints: []string{net.JoinHostPort("127.0.0.1", port)}}})

https://travis-
Looking.
Another instance, or a different thing? https://travis-


Question: t.Fatalf("%s: Expected success, got %v", protoStr, err) } <del> if string(buffer.Bytes()[:len(output)]) != output { t.Errorf("%s: Expected output '%s', got '%v'", protoStr, output, buffer.Bytes()) </del> <ins> if string(buffer.Bytes()) != output { t.Errorf("%s: Expected output '%s', got '%v'", protoStr, output, string(buffer.Bytes())) </ins> } if fcmd.CombinedOutputCalls != 2 {

On some systems, iptables-save outputs the following warning to stderr: Note that the error message includes a newline. Because kube-proxy captures both stdout and stderr when calling , the newline might cause rules to be garbled. For example, the output might contain the following line: The parser treats this as a chain line, and panics, as predicted in
/sig network
FYI
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
Just FYI here's another example of garbled output:
Anyway, fix is in Please review.
/assign
/remove-triage unresolved


Question: klog.V(4).InfoS("Pod cannot start ever", "pod", klog.KObj(update.Options.Pod), "podUID", podUID, "updateType", update.WorkType) return ctx, update, canStart, canEverStart, true case !canStart: <ins> // this is the only path we don't start the pod, so we need to put the change back in pendingUpdate status.pendingUpdate = &update.Options </ins> status.working = false klog.V(4).InfoS("Pod cannot start yet", "pod", klog.KObj(update.Options.Pod), "podUID", podUID) return ctx, update, canStart, canEverStart, true

The following log line is often (~30times) logged during our tests in : . This suggests a possible bug as it was assumed to do not happen. No log line saying . Look at artifacts of all (or almost all) builds of: No response <details</details<detailsReproducible in tests </details<details</details<details</details<details</details<details</details
/sig node FYI
/triage accepted since it's a new error, it is a regression. So adding a corresponding priority /priority important-soon
are you working on this? Should I assign bug to you?
Yes, I'm taking a look at this. /assign
/cc
This is a metrics only error (we don’t report a fraction of metrics) and has no functional impact.
That makes sense, but is it expected that this is a valid state in first place? . My assumption was that if static is false, (i.e. API pod) then we should have a config?
I took a closer look at the logs, it seems that this is happening when we enter while a static pod is being synced which has UID reuse and another pod which has the same full name did not terminate yet. In this case, we block the static pod from starting i.e. () returns false: Because the pod can't start, we don't set : which means that when we log metrics there is no set and thus or are both defaulted to false:
Based on the above, it looks like this case is expected. As a result, maybe the simplest option is to simply drop the log statement since by design we only want to into account a subset of the states for the metrics?


Question: export PATH=${GOPATH}/bin:${PWD}/third_party/etcd:/usr/local/go/bin:${PATH} <del> retry go get github.com/tools/godep && godep version </del> retry go get github.com/jstemmer/go-junit-report # Enable the Go race detector.

Didn't see an issue for this. Godep recently updated its version to v80. This broke several builds that directly godep, grabbing latest. /kind bug cc Example https://k8s- Here are some examples where we grab master:
We have helpers to grab godep. We should use them everywhere.
/cc
/assign


Question: allPorts = append(allPorts, fmt.Sprintf("%v:%v", a.IPProtocol, p)) } } <ins> // Sort all slices to prevent the event from being duped sort.Strings(allPorts) </ins> allow := strings.Join(allPorts, ",") <ins> sort.Strings(fw.SourceRanges) </ins> srcRngs := strings.Join(fw.SourceRanges, ",") <ins> sort.Strings(fw.TargetTags) </ins> targets := strings.Join(fw.TargetTags, ",") return fmt.Sprintf("--description %q --allow %v --source-ranges %v --target-tags %v --project %v", fw.Description, allow, srcRngs, targets, projectID) }

/kind bug /sig network /assign What happened: For events being generated within the GCE cloudprovider, slices of firewall parameters are not being sorted before converted into a string. Due to random ordering, this may result in the same line being repeated.


Question: - apiGroups: - "" resources: <ins> - nodes verbs: - delete - patch - update - apiGroups: - "" resources: </ins> - events verbs: - create

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubelet RBAC Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE Install tools: make quick-release && go run -up What happened: I upgraded the cluster with After which the master node was not ready. What you expected to happen: I expected the master node to be ready. How to reproduce it (as minimally and precisely as possible): Anything else we need to know: In the api server logs: In the kubelet logs:
cc since you were running into this.
cc
c.f. it seems wrong for the node to delete itself
In the case where a kubelet starts up and sees a mismatch between the externalid field and its own idea of what its external id is, it is trying to delete itself. Questions I have: what can make the external id mismatch occur (in cloudprovider and non-cloudprovider cases)? is it even safe for a kubelet to summarily delete itself (as it is trying to do today), or can that mess up apiserver/controller logic around node cleanup, like the attach/detach controller, etc? can we limit the node to reporting the mismatch by updating node status and stay in the tryRegisterWithApiServer loop, and let something like the node controller remove a node that gets in this situation instead? cc
happens in both GCE and GKE upgrades. who can update node status. I would view this as strictly worse than giving kubelets node delete privileges.
If a node instance is recreated with the same name, it will register as the same node. This apparently also happens in tests (). AFAIK, attach/detach controller annotates the node object to keep states. If the node is backed by a new (VM) instance, it is probably safe/better to recreate the node object and start afresh. Of course, upon any node update/delete/add events, the detach/attach controller should examine and reconcile the status. Hm......AFAIK node controller also does not coordinate with other controllers (e.g., detach/attach). I am not sure how this would help the second point you made.

would you be ok with adding delete permissions for kubelet because it has them and uses them today, and then taking the argument about whether it should be done another way to it's own issue?
yeah, convinced me that if the node is the authority of the spec during self-registration cases, update/delete permissions are reasonable, and we'd need to figure out how to scope those to update/delete self in the future, just like we do for update status
see


Question: pod: &api.Pod{ Spec: api.PodSpec{ InitContainers: []api.Container{{ <ins> Name: "dummy", Resources: api.ResourceRequirements{ Requests: api.ResourceList{api.ResourceCPU: resource.MustParse("1m")}, Limits:   api.ResourceList{api.ResourceCPU: resource.MustParse("2m")}, }, }}, }, }, required: []corev1.ResourceName{corev1.ResourceMemory}, err:      `must specify memory for: dummy`, }, "multiple init container resource missing": { pod: &api.Pod{ Spec: api.PodSpec{ InitContainers: []api.Container{{ Name: "foo", Resources: api.ResourceRequirements{ Requests: api.ResourceList{api.ResourceCPU: resource.MustParse("1m")}, Limits:   api.ResourceList{api.ResourceCPU: resource.MustParse("2m")}, }, }, { Name: "bar", </ins> Resources: api.ResourceRequirements{ Requests: api.ResourceList{api.ResourceCPU: resource.MustParse("1m")}, Limits:   api.ResourceList{api.ResourceCPU: resource.MustParse("2m")},

I deployed a pod with and on a namesapace with a set with and elements and I get the error: The pod to be deployed Create a new namespace Create (using a manifest file named resource-quota-) and verify a named compute-resources Content of the resource-quota- file Create (using a manifest file named ) a pod named high-priority with the and for both and . Content of the file The pod is deployed correctly if I delete the <details</details<detailsaws </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig Scheduling
/wg API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg API Expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg <API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg <API ExpressionInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg "API Expression"
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg "API Expression" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg APIExpression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg APIExpression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/wg wg-api-expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg wg-api-expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hey, v1.18.20 is out of support now, you should try with a newer version. Besides, I tried with an old version v1.18.19, but worked as well.
Are u using some mutating webhook which inject some sidecar into the pod without setting resources fields?
It's probably that, but IDK which WebHook is doing that. The feature you suggested would really help.
hi the fix pr has been merged, maybe u could use it to locate which container is causing the problem.
Thanks


Question: Logf("Waiting up to %v for all pods (need at least %d) in namespace '%s' to be running and ready", timeout, minPods, ns) if wait.Poll(poll, timeout, func() (bool, error) { <del> // We get the new list of pods in every iteration beause more pods come // online during startup and we want to ensure they are also checked. </del> <ins> // We get the new list of pods and replication controllers in every // iteration because more pods come online during startup and we want to // ensure they are also checked. rcList, err := c.ReplicationControllers(ns).List(labels.Everything()) if err != nil { Logf("Error getting replication controllers in namespace '%s': %v", ns, err) return false, nil } replicas := 0 for _, rc := range rcList.Items { replicas += rc.Spec.Replicas } </ins> podList, err := c.Pods(ns).List(labels.Everything(), fields.Everything()) if err != nil { Logf("Error getting pods in namespace '%s': %v", ns, err) return false, nil } <del> nOk, badPods := 0, []api.Pod{} </del> <ins> nOk, replicaOk, badPods := 0, 0, []api.Pod{} </ins> for _, pod := range podList.Items { if res, err := podRunningReady(&pod); res && err == nil { nOk++ <ins> if hasReplicationControllersForPod(rcList, pod) { replicaOk++ } </ins> } else { <del> badPods = append(badPods, pod) </del> <ins> if pod.Status.Phase != api.PodFailed { Logf("The status of Pod %s is %s, waiting for it to be either Running or Failed", pod.ObjectMeta.Name, pod.Status.Phase) badPods = append(badPods, pod) } else if !hasReplicationControllersForPod(rcList, pod) { Logf("Pod %s is Failed, but it's not controlled by a ReplicationController", pod.ObjectMeta.Name) badPods = append(badPods, pod) } //ignore failed pods that are controlled by a replication controller </ins> } } Logf("%d / %d pods in namespace '%s' are running and ready (%d seconds elapsed)", nOk, len(podList.Items), ns, int(time.Since(start).Seconds())) <del> if nOk == len(podList.Items) && nOk >= minPods { </del> <ins> Logf("expected %d pod replicas in namespace '%s', %d are Running and Ready.", replicas, ns, replicaOk) if replicaOk == replicas && nOk >= minPods && len(badPods) == 0 { </ins> return true, nil } logPodStates(badPods)

On a healthy test cluster the health check looks like this: On a failed cluster it looks like this:
Possibly related to ?
this has elasticsearch logging failing.
As per discussion with I think that the solution here is to ignore failed pods for these test cluster health checks. The failed pods were all correctly restarted by the relevant replication controllers, so the tests should have been able to proceed in this and all the other similar cases that we looked at. Why the pods failed is a separate question, which will hopefully be detected and fixed elsewhere.
It might be due to
Just waiting for oncall to merge