Question: importpath = "k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/proxy", deps = [ "//cmd/kubeadm/app/apis/kubeadm:go_default_library", <ins> "//cmd/kubeadm/app/constants:go_default_library", </ins> "//cmd/kubeadm/app/util:go_default_library", "//cmd/kubeadm/app/util/apiclient:go_default_library", "//pkg/proxy/apis/kubeproxyconfig/scheme:go_default_library",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After joining a new ppc64le node to the k8s cluster, the kube-proxy DaemonSet fails on the new node. What you expected to happen: It was expected that the kube-proxy DS would "just work" as it does when adding an amd64 node to the cluster. How to reproduce it (as minimally and precisely as possible): Set up an amd64 based k8s cluster, then join a ppc64le node (or any non-amd64 arch) to the cluster. See the pod fail via 'kubectl -n kube-system get pods' because it's pulling the amd64 kube-proxy docker image instead of the ppc64le image. Anything else we need to know?: The yaml for the DaemonSet specifies the image: <versionIt would be preferable to have a manifest so that the image in the DS could be specified as <versionand could "just work" on any platform. See: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: self - cluster installed via kubeadm OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.3 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTU_CODENAME=xenial Kernel (e.g. ): N/A Install tools: N/A Others: N/A
if you are running multi-architecture nodes, your daemonsets should specify a node selector to limit their arch-specific image to run on matching nodes
/close
I am well aware that a node selector can be used. That is what we are doing now. But it is a hack. The default should "just work". Adding a new non-amd64 node should not result in kube-proxy immediately failing on it because the default daemon set is amd64 arch specific. If kube-proxy is meant to "just work" clusterwide, a node selector should not be needed.

Please use the manifest-tool so that the default "just works". Please see the weave-net plugin. They are using the manifest-tool and it "just works" on amd64 and ppc64le. Their daemon set does not require/use a nodeSelector.
The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this
Reiterating the mentions to trigger a notification: <detailsIn response to : The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this </details
ah, this was a comment about the kube-proxy image, not the daemonset functionality /reopen
the kube-proxy addon daemonset is probably the multi-arch kube-proxy image to support that daemonset might be
Looks like it's coming from here: /assign /sig cluster-lifecycle Lucas, Do we want to avoid the references?
A dirty workaround for this is to pull and rename to . The kube-proxy is running as daemonset, created during . That means the kube-proxy image arch keeps consistent with your master node. Currently we don't have a multi-arch kube-proxy image yet. We have to explicitly specify the right name, like .
Hi .. Yes, we just created a separate daemonset with a ppc64le node slector that refers to the ppc64le image directly. But it's a bit of a hack. That is why I opened this ticket looking for a multi-arch image. Thanks!
I don't know that mixed-ISA clusters are supported. cc
The requirement is: only use "" in DaemonSet yaml file, and the k8s will download the correct image according to the ARCH info. It's more about image registry's feature; and it seems some registries had supported that: .
Yeah, has already supported multi-arch. But actually all the images now are architecture specific, like and . We've not built any multi-arch images right now.
for the test images we have the ball rolling here - - so if you can do the same (or similar) for other images, that would be great!
I have pending PR in releases project for creating manifest for those -
The manifest images created in kubernetes/release should work for future released versions, but will they be created during CI testing?

if you see one of the jobs that pushes images say you can grep for that's basically from the same files that got changed in so i fully expect the images pushed to CI will reflect the changes made in 516.
please confirm that i am reading this right ^ :)
I was more concerned with ci jobs that do not use cross builds, such as https://k8s-
please open an issue about pushing manifests in CI as well


Question: // 'MemoryPressure', 'OutOfDisk' and 'DiskPressure'. TaintNodesByCondition utilfeature.Feature = "TaintNodesByCondition" <del> // owner: @haibinxie // alpha: v1.8 // // Implement IPVS-based in-cluster service load balancing SupportIPVSProxyMode utilfeature.Feature = "SupportIPVSProxyMode" </del> // owner: @jsafrane // alpha: v1.8 //

There are some work to be done in the follow-up of , that is, help IPVS existing alpha. [x] GCE CI failures()[x] cross-node visit NodePort ()[x] hairpin issue ()[x] Using ipset doing snat and packet filtering and reduce iptables call().[x] automatically load kernel modules. (, )[x] creating e2e test suite for IPVS in upstream CI ().[x] ip cmd -netlink bind/unbind IP address to/from dummy device()[x] rsync to HEAD of iptables(, )[x] enable flush batch API()[x] Fix TODOs in codes ()[x] Bug report (, )[x] Default sync period()[x] IPV6 support(, and more...) /cc
/cc
ref ! It will generate annoy error info If unbind multi-ports on one service .
You're right. I think it's a nice finding. Can you fire a PR to fix it?
Sure, I'll make it soon.
I found it is treated as already bound if returned 2 in , but ref to the and of ip , it just means an error was reported by the kernel, such as "Operation not permitted". I prefer to use instead of command to manage ip address.
Actually, I have considered to use netlink before. I think what you said make sense, but can we discuss it in other thread? I mean, create a new issue and link it here? Thanks!
I opened a issue
on the subject :)
OOPS... I will fix that typo. Thanks
Wish to use ipset to replace KUBE-MARK-MASQ chain.
I find it's a reasonable suggestion and am trying to make it workable...
There are to ways to add IPVS CI job: , but doesn't support any kube-proxy config right now, so I had this pr , and also, need support it too, see this pr: kubernetes/kubernetes-anywhere build cluster. See this pr: kubernetes/test-infra The second way is much easier and faster, but I was told they gonna deprecated the , so in the long term, we still need and 's support. If anybody have better way or have some suggestion about this, please let me know, I'll be really appreciate, Thanks.
This one is done as both PRs are merged:
ACK!

/area ipvs
We are almost reaching the beta, except GCE CI failures, see The failure is the case that, visiting Cluster IP from Pod A, and the backend Pod B is in the same host with the Pod A. Pod A failed to get response from Pod B until timeout. I can reproduce the failures in my GCE cluster with kubent as network plugin, but it works fine in my local-up cluster and people has never reported the issue yet. and I are digging...
IPVS CI is green now.


Question: if len(o.ConfigFile) == 0 && len(o.WriteConfigTo) == 0 { klog.Warning("WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.") o.applyDeprecatedHealthzPortToConfig() <ins> o.applyDeprecatedMetricsPortToConfig() </ins> } // Load the config file here in Complete, so that Validate validates the fully-resolved config.

What happened: From We used to allow using or in and . Now the validation on kubeproxyconfig disallow that (it only allows ): This seems to break backward compatibility for those two flags. Please comment if I'm missing something. cc How to reproduce it (as minimally and precisely as possible): Try with kube-proxy. Environment: Kubernetes version (use ):
/sig network /kind bug
Tried , the error is: and , the error is: So maybe we need to loosen and revert ?
Loosen the validation sounds good. was meant for loosen the validation as well (allowing not just ip, but also ip:port), but for some reason it doesn't work as intended. Would be great to fix that instead of revert :)
find the root cause of not working as expected. introduced the regression: I send a PR to fix the two issues we mentioned in this thread.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .

/close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I believe this hasn't been fixed yet. /remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : I believe this hasn't been fixed yet. /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I agree There's also the issue of IPv6-only: Results in:
Thanks for your report. Do you mind opening a new issue regarding IPv6 since the original issue was fixed? Thanks.
Sure. Thanks
Please see:


Question: expectedList := &unstructured.UnstructuredList{} expectedList.SetGroupVersionKind(listGVK) expectedList.SetResourceVersion("") // by product of the fake setting resource version <ins> expectedList.SetContinue("") </ins> expectedList.Items = append(expectedList.Items, u) if diff := cmp.Diff(expectedList, list); diff != "" {

The fake dynamic client does not support mock pagination or the self link. Add ability to propagate the metadata fields from a mock unstructured list to the served list by the fake lister. Repro linked in the MR x-ref. No response <details</detailsN/A N/A <detailsN/A </details<detailsN/A </details<detailsN/A </details
/sig api-machinery
/cc /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: Corefile: | .:53 { errors <del> log stdout </del> <ins> log </ins> health <del> kubernetes $DNS_DOMAIN $SERVICE_CLUSTER_IP_RANGE </del> <ins> kubernetes $DNS_DOMAIN $SERVICE_CLUSTER_IP_RANGE { pods insecure } </ins> prometheus proxy . /etc/resolv.conf cache 30

Is this a BUG REPORT or FEATURE REQUEST?: BUG What happened: When CoreDNS is deployed by default using feature-gates of kubeadm, it does not log anything when processing a DNS query. That is due to change of log option of CoreDNS in latest version. that version was bumped up in kubeadm, but the manifest was not adapted to the new log option. initial change of 'log stdout' in CoreDNS : version of CoreDNS 1.0.0 bumped into Kubeadm : NOTE: I guess we should also changed manifests for kube-up : see The change in manifest expected is to remove the option "stdout" in the line "log ...". As it is now interpreted as a scope of domain to log. Invalid manifest: Environment: Kubernetes version (use ): 1.9 and MAIN Cloud provider or hardware configuration: any Install tools: kubeadm, kube-up the configmap for coredns kubectl get configmaps -n kube-system coredns -o yaml kubectl replace -f - the coredns deployment kubectl scale deployment --replicas=0 -n kube-system coredns kubectl scale deployment --replicas=1 -n kube-system coredns
/cc /sig network
NOTE: only log is impacted. CoreDNS working as expected.
/cc
I have a question : the initial manifest is done for the image 0.9.10 of CoreDNS. Bumping the version of CoreDNS was done by changing the version in file (see ) which means that the image's version is STILL 0.9.10 in the manifests ... and the corefile config is correct against the version of the manifest. QUESTION: when modifying the , is there a way to upgrade the manifest by program ? I guess at the time the new version number is applied on the manifest ...
thanks for reporting this! I send PR to update the Corefile. I don't think of a way so far. I guess we can't unless there is a way to get the Corefile from some place of upstream (CoreDNS).


Question: return 0, nil } <del> // CleanupAndExit remove iptables rules and exit if success return nil </del> <ins> // CleanupAndExit remove iptables rules and ipset/ipvs rules in ipvs proxy mode // and exit if success return nil </ins> func (s *ProxyServer) CleanupAndExit() error { <del> encounteredError := userspace.CleanupLeftovers(s.IptInterface) encounteredError = iptables.CleanupLeftovers(s.IptInterface) || encounteredError encounteredError = ipvs.CleanupLeftovers(s.IpvsInterface, s.IptInterface, s.IpsetInterface, s.CleanupIPVS) || encounteredError </del> <ins> // cleanup IPv6 and IPv4 iptables rules ipts := []utiliptables.Interface{ utiliptables.New(s.execer, utiliptables.ProtocolIPv4), utiliptables.New(s.execer, utiliptables.ProtocolIPv6), } var encounteredError bool for _, ipt := range ipts { encounteredError = userspace.CleanupLeftovers(ipt) || encounteredError encounteredError = iptables.CleanupLeftovers(ipt) || encounteredError encounteredError = ipvs.CleanupLeftovers(s.IpvsInterface, ipt, s.IpsetInterface, s.CleanupIPVS) || encounteredError } </ins> if encounteredError { return errors.New("encountered an error while tearing down rules") }

What happened: Currently kube-proxy cleanup not work, which will left iptables (and ipvs) rules in host and. What you expected to happen: Kube-proxy cleanup can work and not relay on hostName or ClientConnection config. How to reproduce it (as minimally and precisely as possible):
/sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


Question: return nil } <del> // logWatchChannelErr checks whether the error is about mvcc revision compaction which is regarded as warning </del> func logWatchChannelErr(err error) { <del> if !strings.Contains(err.Error(), "mvcc: required revision has been compacted") { klog.Errorf("watch chan error: %v", err) } else { </del> <ins> switch { case strings.Contains(err.Error(), "mvcc: required revision has been compacted"): // mvcc revision compaction which is regarded as warning, not error </ins> klog.Warningf("watch chan error: %v", err) <ins> case isCancelError(err): // expected when watches close, no need to log default: klog.Errorf("watch chan error: %v", err) </ins> } }

TestChangeCRD " TestChangeCRD": 37, Recently, not sure the date. No response_ /sig api-machinery
also flaking in weirdly, this doesn't show up at all in
cc I can't tell if this is related to the openapi change or not
Actually, I think this is an issue either with the test or with watch behavior, not sure which. The test starts 10 watches, lets them run, then waits for them to exit cleanly, and instead, they are getting watch errors. I can't tell if the watch errors are expected or not, or if server or watch client behavior changed here or not. Opened to debug further
and stand out as a recent change to watching
cc on whether the watching changes in could explain this integration test getting watch errors it didn't previously get
also stands out
would explain this integration test getting watch errors it didn't previously get?
the filter is enabled only when the server run option has a positive time duration (the default is zero). I searched the k/k repo, only the unit tests in enables this filter: I did a quick scan of the logs, did not see any graceful termination activity either. I will take a deeper look at the logs
and all merged changes were limited to that filter?
This is a Go test, so the entire test output is treated as the failure message. That output is very different from one failure to the other, so in triage we probably have x different "clusters" with one failure in each, instead of one cluster with x failures. I don't know what triage does with such small clusters.
we have changes in only the the filter adds to the request context, I just searched the code base, did not find any other call site for by default, should block (on a nil channel) If the test is receiving an event, it implies that the following case is executing, right?
xref: ??
Thank you all for deflaking.
/triage accepted


Question: utilfs "k8s.io/kubernetes/pkg/kubelet/kubeletconfig/util/filesystem" ) <ins> const kubeletFile = "kubelet" </ins> // Loader loads configuration from a storage layer type Loader interface { // Load loads and returns the KubeletConfiguration from the storage layer, or an error if a configuration could not be loaded

From in The most important things I want to see through node e2e tests: 1) the kubelet is started from last-known-good-config file 2) kubelet can download the new config properly to the node; and restart itself from the new config file 3) kubelet can mark the config bad properly after several round of crashing caused the bad config 4) Kubelet can roll back the last-known-good-config and restart itself from that good one 5) kubelet can properly reports its configuration status Note: (2) is covered by existing e2enode tests (1, 4, 5) were covered by the , all of which I plan to turn into automated tests after this PR merges - but I'm trying not to increase the size of this PR any further ~(3) we need to figure out how to test this. I don't want to rely on a "valid" config that causes a crash loop - because the very existence of such a config should be viewed as a bug. I also don't want to include a "please crash loop for our tests" field in the config, because that's an ugly API. The best solution is not clear here.~ We decided to remove crash-loop detection, see . Filing this issue to track this work: into e2e_node tests [x] remote config [x] local (from file) config [x] ~test the crash-loop detection~ removed, see above /cc
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.8 milestone for more than 7 days: kind: Must specify exactly one of [, , ]. priority: Must specify exactly one of [, , ]. Removing it from the milestone. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
The test cases in the OP have been covered by our e2e_node tests for a while.


Question: pluginNames = a.enabledPluginNames() } <del> pluginsConfigProvider, err := admission.ReadAdmissionConfiguration(pluginNames, a.ConfigFile, scheme) </del> <ins> pluginsConfigProvider, err := admission.ReadAdmissionConfiguration(pluginNames, a.ConfigFile, configScheme) </ins> if err != nil { return fmt.Errorf("failed to read plugin config: %v", err) }

/kind bug What happened: When I want to enable both plugins by passing below flag, as they both need config file, and I find I donot know what to set If I set , then I get an error in apiserver
/kind bug /sig auth
/cc
/cc
first, you can't put comments in json. the error was complaining about your embedded comments. this would let the server start: second, the docs are really, really, not right (opened with fixes). if you want to specify config for multiple plugins, should point to a config that references individual config files for each plugin (which there's a bug in loading... fixed in ): example
/assign


Question: framework.Failf("%v", err) } <del> ginkgo.By("kubectl explain works for CR with the same resource name as built-in object") if err := verifyKubectlExplain(f.Namespace.Name, customServiceShortName+".spec", `(?s)DESCRIPTION:.*Specification of CustomService.*FIELDS:.*dummy.*<string>.*Dummy property`); err != nil { framework.Failf("%v", err) } </del> ginkgo.By("kubectl explain works to return error when explain is called on property that doesn't exist") if _, err := framework.RunKubectl(f.Namespace.Name, "explain", crd.Crd.Spec.Names.Plural+".spec.bars2"); err == nil || !strings.Contains(err.Error(), `field "bars2" does not exist`) { framework.Failf("unexpected no error when explaining property that doesn't exist: %v", err)

Which jobs are failing: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) Which test(s) are failing: [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema [Conformance] Since when has it been failing: 04-26 05:02 PDT Testgrid link: Reason for failure: Apr 26 12:28:01.219: FAIL: failed to explain error running /workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://35.185.126.243 -- explain Command stdout: stderr: the server doesn't have a resource type "ksvc" error: exit status 1 Full Stack Trace () +0x1545 (0xc002ca6100) +0x324 (0xc002ca6100) +0x2b testing.tRunner(0xc002ca6100, 0x4b8d060) +0xc9 created by testing.(T).Run +0x350 [AfterEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] ï¿½[1mSTEPï¿½[0m: Collecting events from namespace "crd-publish-openapi-4377". ï¿½[1mSTEPï¿½[0m: Found 0 events. Apr 26 12:28:01.314: INFO: POD NODE PHASE GRACE CONDITIONS Apr 26 12:28:01.314: INFO: Apr 26 12:28:02.899: INFO: skipping dumping cluster info - cluster too large Apr 26 12:28:02.899: INFO: Waiting up to 3m0s for all (but 50) nodes to be ready ï¿½[1mSTEPï¿½[0m: Destroying namespace "crd-publish-openapi-4377" for this suite. Anything else we need to know: /cc /cc /milestone v1.19 /priority critical-urgent
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

<detailsIn response to : Which jobs are failing: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema [Conformance] 04-26 05:02 PDT Apr 26 12:28:01.219: FAIL: failed to explain error running /workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://35.185.126.243 -- explain Command stdout: the server doesn't have a resource type "ksvc" exit status 1 () (0xc002ca6100) (0xc002ca6100) testing.tRunner(0xc002ca6100, 0x4b8d060) created by testing.(T).Run [AfterEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] ï¿½[1mSTEPï¿½[0m: Collecting events from namespace "crd-publish-openapi-4377". ï¿½[1mSTEPï¿½[0m: Found 0 events. Apr 26 12:28:01.314: INFO: POD NODE PHASE GRACE CONDITIONS Apr 26 12:28:01.314: INFO: Apr 26 12:28:02.899: INFO: skipping dumping cluster info - cluster too large Apr 26 12:28:02.899: INFO: Waiting up to 3m0s for all (but 50) nodes to be ready ï¿½[1mSTEPï¿½[0m: Destroying namespace "crd-publish-openapi-4377" for this suite. /cc /milestone v1.19 </details
/milestone v1.19 Thanks for opening this! I'll get you on milestone maintainers soon :) Also, was taking a look at this as well and noticed this test was recently :
As noted in , this modified an existing conformance test and made it flaky. Please move the new test out of the conformance test and mark it as flaky until this issue has been resolved.

Cc for flake introduced in
It is a bit weird that I cannot reproduce this problem on my local kind cluster by running even after 20 times I suspect this is a cache issue since I once created a crd without short name and forgot to delete it in the test.
/retitle [Flaky Test]: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) /kind flake /remove-kind failing-test
/assign
I can reproduce this problem by the following steps now. It looks like that this problem has something to do with discovery cache. <pre$ cat <summary<detailsapiVersion: kind: CustomResourceDefinition metadata: name: spec: conversion: strategy: None group: crd-publish-openapi-test- names: kind: Service listKind: ServiceList plural: services singular: service shortNames: ["ksvc"] scope: Namespaced versions: name: v1alpha1 schema: openAPIV3Schema: properties: spec: description: Specification of CustomService properties: dummy: description: Dummy property. type: string type: object type: object served: true storage: true </details</summary$ kubectl get crd No resources found in default namespace. $ rm -r $ kubectl explain the server doesn't have a resource type "ksvc" $ kubectl create -f created $ kubectl explain the server doesn't have a resource type "ksvc" $ kubectl explain <summary<detailsKIND: Service VERSION: crd-publish-openapi-test- RESOURCE: spec <ObjectDESCRIPTION: Specification of CustomService FIELDS: dummy <stringDummy property. </details</summary</pre
looks like
/assign


Question: -o "${tmp_kube_master_certs}"  http://metadata.google.internal/computeMetadata/v1/instance/attributes/kube-master-certs # Convert the yaml format file into a shell-style file. <del> eval $(python -c ''' </del> <ins> eval $(${PYTHON} -c ''' </ins> import pipes,sys,yaml for k,v in yaml.load(sys.stdin).iteritems(): print("readonly {var}={value}".format(var = k, value = pipes.quote(str(v))))

What happened: Pick an OS image where python is python3 and the cluster will not come up. What you expected to happen: Cluster should come up. How to reproduce it (as minimally and precisely as possible): Switch the default OS image to use an OS which has python3 as the default python. Run cluster/kube- for GCE. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/assign


Question: <ins> #Service Accounts ## Motivation Processes in Pods may need to call the Kubernetes API.  For example: - scheduler - replication controller - minion controller - a map-reduce type framework which has a controller that then tries to make a dynamically determined number of workers and watch them - continuous build and push system - monitoring system They also may interact with services other than the Kubernetes API, such as: - an image repository, such as docker -- both when the images are pulled to start the containers, and for writing images in the case of pods that generate images. - accessing other cloud services, such as blob storage, in the context of a larged, integrated, cloud offering (hosted or private). - accessing files in an NFS volume attached to the pod ## Design Overview A service account binds together several things: - a *name*, understood by users, and perhaps by peripheral systems, for an identity - a *principal* that can be authenticated and (authorized)[../authorization.md] - a [security context](./security_contexts.md), which defines the Linux Capabilities, User IDs, Groups IDs, and other capabilities and controls on interaction with the file system and OS. - a set of [secrets](./secrets.md), which a container may use to access various networked resources. ## Design Discussion A new object Kind is added: ```go type ServiceAccount struct { TypeMeta   `json:",inline" yaml:",inline"` ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"` username string securityContext ObjectReference // (reference to a securityContext object) secrets []ObjectReference // (references to secret objects } ``` The name ServiceAccount is chosen because it is widely used already (e.g. by Kerberos and LDAP) to refer to this type of account.  Note that it has no relation to kubernetes Service objects. The ServiceAccount object does not include any information that could not be defined separately: - username can be defined however users are defined. - securityContext and secrets are only referenced and are created using the REST API. The purpose of the serviceAccount object is twofold: - to bind usernames to securityContexts and secrets, so that the username can be used to refer succinctly in contexts where explicitly naming securityContexts and secrets would be inconvenient - to provide an interface to simplify allocation of new securityContexts and secrets. These features are explained later. ### Names From the standpoint of the Kubernetes API, a `user` is any principal which can authenticate to kubernetes API. This includes a human running `kubectl` on her desktop and a container in a Pod on a Node making API calls. There is already a notion of a username in kubernetes, which is populated into a request context after authentication. However, there is no API object representing a user.  While this may evolve, it is expected that in mature installations, the canonical storage of user identifiers will be handled by a system external to kubernetes. Kubernetes does not dictate how to divide up the space of user identifier strings.  User names can be simple Unix-style short usernames, (e.g. `alice`), or may be qualified to allow for federated identity ( `alice@example.com` vs `alice@example.org`.)  Naming convention may distinguish service accounts from user accounts (e.g. `alice@example.com` vs `build-service-account-a3b7f0@foo-namespace.service-accounts.example.com`), but Kubernetes does not require this. Kubernetes also does not require that there be a distinction between human and Pod users.  It will be possible to setup a cluster where Alice the human talks to the kubernetes API as username `alice` and starts pods that also talk to the API as user `alice` and write files to NFS as user `alice`.  But, this is not recommended. Instead, it is recommended that Pods and Humans have distinct identities, and reference implementations will make this distinction. The distinction is useful for a number of reasons: - the requirements for humans and automated processes are different: - Humans need a wide range of capabilities to do their daily activities. Automated processes often have more narrowly-defined activities. - Humans may better tolerate the exceptional conditions created by expiration of a token.  Remembering to handle this in a program is more annoying.  So, either long-lasting credentials or automated rotation of credentials is needed. - A Human typically keeps credentials on a machine that is not part of the cluster and so not subject to automatic management.  A VM with a role/service-account can have its credentials automatically managed. - the identity of a Pod cannot in general be mapped to a single human. - If policy allows, it may be created by one human, and then updated by another, and another, until its behavior cannot be attributed to a single human. **TODO**: consider getting rid of separate serviceAccount object and just rolling its parts into the SecurityContext or Pod Object. The `secrets` field is a list of references to /secret objects that an process started as that service account should have access to to be able to assert that role. The secrets are not inline with the serviceAccount object.  This way, most or all users can have permission to `GET /serviceAccounts` so they can remind themselves what serviceAccounts are available for use. Nothing will prevent creation of a serviceAccount with two secrets of type `SecretTypeKubernetesAuth`, or secrets of two different types.  Kubelet and client libraries will have some behavior, TBD, to handle the case of multiple secrets of a given type (pick first or provide all and try each in order, etc). When a serviceAccount and a matching secret exist, then a `User.Info` for the serviceAccount and a `BearerToken` from the secret are added to the map of tokens used by the authentication process in the apiserver, and similarly for other types.  (We might have some types that do not do anything on apiserver but just get pushed to the kubelet.) ### Pods The `PodSpec` is extended to have a `Pods.Spec.ServiceAccountUsername` field.  If this is unset, then a default value is chosen.  If it is set, then the corresponding value of `Pods.Spec.SecurityContext` is set by the Service Account Finalizer (see below). TBD: how policy limits which users can make pods with which service accounts. ### Authorization Kubernetes API Authorization Policies refer to users.  Pods created with a `Pods.Spec.ServiceAccountUsername` typically get a `Secret` which allows them to authenticate to the Kubernetes APIserver as a particular user.  So any policy that is desired can be applied to them. A higher level workflow is needed to coordinate creation of serviceAccounts, secrets and relevant policy objects. Users are free to extend kubernetes to put this business logic wherever is convenient for them, though the Service Account Finalizer is one place where this can happen (see below). ### Kubelet The kubelet will treat as "not ready to run" (needing a finalizer to act on it) any Pod which has an empty SecurityContext. The kubelet will set a default, restrictive, security context for any pods created from non-Apiserver config sources (http, file). Kubelet watches apiserver for secrets which are needed by pods bound to it. **TODO**: how to only let kubelet see secrets it needs to know. ### The service account finalizer There are several ways to use Pods with SecurityContexts and Secrets. One way is to explicitly specify the securityContext and all secrets of a Pod when the pod is initially created, like this: **TODO**: example of pod with explicit refs. Another way is with the *Service Account Finalizer*, a plugin process which is optional, and which handles business logic around service accounts. The Service Account Finalizer watches Pods, Namespaces, and ServiceAccount definitions. First, if it finds pods which have a `Pod.Spec.ServiceAccountUsername` but no `Pod.Spec.SecurityContext` set, then it copies in the referenced securityContext and secrets references for the corresponding `serviceAccount`. Second, if ServiceAccount definitions change, it may take some actions. **TODO**: decide what actions it takes when a serviceAccount defintion changes.  Does it stop pods, or just allow someone to list ones that out out of spec?  In general, people may want to customize this? Third, if a new namespace is created, it may create a new serviceAccount for that namespace.  This may include a new username (e.g. `NAMESPACE-default-service-account@serviceaccounts.$CLUSTERID.kubernetes.io`), a new securityContext, a newly generated secret to authenticate that serviceAccount to the Kubernetes API, and default policies for that service account. **TODO**: more concrete example.  What are typical default permissions for default service account (e.g. readonly access to services in the same namespace and read-write access to events in that namespace?) Finally, it may provide an interface to automate creation of new serviceAccounts.  In that case, the user may want to GET serviceAccounts to see what has been created. </ins>

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
/assign /area kubeadm
/sig cluster-lifecycle
i went trough all the flakes we have at the moment and adjusted a number timeouts. here is the PR:
was green for a bit since merge, but noticed a timeout on this morning. Do you think it is worth bumping the timeout there as well? It looks like the timeout was quite a large outlier compared to the successful runs (more than double the duration), so maybe we should address underlying problem.
lets wait for some more singlal before bumping the timeouts on init. locally for me takes a couple of minutes, so i would like to know if this is a temporary GCE/networking issue.
actually something is not right, because in this job: also this job showed that even 10m is not enough for it errored with: but looking at the log timestamps only ~2 minutes have passed. i will have another look later today or tomorrow.
/retitle [Flaky Test] kubeadm-kinder-upgrade-1-18-master (ci-kubernetes-e2e-kubeadm-kinder-upgrade-1-18-master) /priority important-soon /remove-kind failing-test /kind flake the test seems to be timeouting occasionally now. this appeared to be a temporary network speed issue. in the meantime i have created this PR which attempts to catch network flakes early and reduce overall network traffic:
the kubeadm e2e jobs started requiring more than 40 minutes: https://k8s- so a lot of test jobs are now failing. this could be due to: resource limits in Prow more conformance tests were during the weekend or next week i will try bumping all jobs to ~60 minutes.
/remove-priority important-soon (prio is already critical)
xref
hi, the jobs seem to be now at the state where they were before the ticket was logged - i.e. mostly green. the Prow issue that was discussed in is definitely a factor, so i'm not going to bump the jobs to 60minutes for now.

we do still see 2 occasional types of flakes: one of them is known and we don't have a fix for it as it's complicated. for the other one i'm going to send a PR and ping you for a quick lgtm. thanks.
thanks for following up! I will make sure to take a look at your incoming PR :+1:
i haven't seen the same failure in the last 10 runs: https://k8s- please re-open this ticket if needed. thanks.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am seeing some pretty regular failures here again. Could you take a look? /reopen
Reopened this issue. <detailsIn response to : I am seeing some pretty regular failures here again. Could you take a look? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the job is taking more than 40 minutes. so this is related to:
The runs in last few days have been well under 60m, except for
i'm suspecting slow CPU/network for some of the recent failures.
/remove-sig testing SIG testing does not own kinder or upgrades
looks like the most prominent recent flakes are caused by the inability to prepare the node image or upgrade all nodes in less than 10 minutes. i will send a PR to increase these timeouts.
Is that test now named ?
yes. i see some occasional flakes of not being able to complete the conformance suite in 25 minutes (parallel).
/retitle [Flaky Test] kubeadm-kinder-upgrade-1-18-1-19 (ci-kubernetes-e2e-kubeadm-kinder-upgrade-1-18-1-19)
i can conform that the suite requires mostly more than 25 minutes in later releases (either due to more tests or slower infra). this PR increases the e2e test suite in kubeadm to 35 minutes and will close this ticket: we should create new / separate tickets with flaky tasks in the job.


Question: if options.Limit == 0 { options.Limit = p.PageSize } <ins> requestedResourceVersion := options.ResourceVersion </ins> var list *metainternalversion.List for { select {

/sig scalability Debugging below done by Failed test attempt: It failed with a number of "WaitingForPods: objects timed out" errors (46 failed objects in total). Sample failing pods: 580 nodes were not ready at 18:37: nodelifecyclecontroller said that gce-scale-cluster-minion-group-2-66d8's status hasn't been updated for 41 seconds. apiserver's logs show that kubelet was doing 'PUT ' every 10 seconds. Something weird happened to watch by 'shared-informers': One watch started at 18:36:52 and finished 9ms later. The next one started at 18:40:34. So it seems that there was a gap in lease watche between 18:36:52 and 18:40:34. Between 18:36:52 and 18:40:34 there is a number of leases LIST requests in the following pattern: a single /leases?limit=500 LIST request that finishes with 200 a single /leases/continue=XXX request that finishes with 400 The kube-controller-manager is retrying every 1 second up to 18:40:34.
/assign
Looking further at this it seems that number of LIST leases by kube-controller-manager increased significantly ~week ago in large scale tests (both gce and kubemark). GCE: ! Kubemark: ! This was caused by which as a side effect enabled pagination of LIST calls when starting/restarting watchers. In some cases we're getting 400 errors from api-server for this calls which results in the symptoms described above, i.e nodes becoming not-ready and pods stuck in running but not ready state for some time. GCE: ! Kubemark: !
There are actually two issues here: that we switched from non-paginated LIST calls to paginated the 10x increase in # calls is expected, but what is worrying is that we already had ~50 LISTs in every test. Such LIST happens only when watcher restarts which shouldn't happen in a healthy cluster This will be address in need to figure out why we're sometimes getting 400 response (BAD REQUEST).

/milestone v1.17
Logs when then 400 errors occurred api-server kube-controller-manager
So what happens here is: in reflector ends with an error different than so the doesn't get reset to next list is issued with set to page is fetched successfully, the next page is requested with the got from the first page and second page fails with due to error the doesn't get cleared anywhere so we go back to 2 and the 400 error keeps repeating until we get 410 (resourceVersionTooOld, on the next etcd compaction) and the is set to takes long enough for the NodeController to mark many nodes as not ready as it doesn't get any updates from watch about node leases This was indeed introduced by but the actual bug is in the which sets both and on the subsequent pages while it should set only . I will send a PR to fix it today.
FYI
Great finding.
Does this mean we have no tests that actually check if a reflector works with paging?
I see, it's the rare situation where "Watch in reflector ends with an error different than 410" that is untested. Can we set up a unit test that exercises this scenario by wrapping etcd3 storage to allow injecting an error in a watch attempt (similar to how we wrapped storage to test stale behavior in )?
I don't think it's the case - see . It clearly seems to be that pager wasn't properly tested. The problem seems to be that we're testing it with artifical stuff, whereas in real life etcd would return "bad request" errors.
Right, I'd like to see if we can trigger the error condition in a way that the restart/paging requests go to a real storage backend.
/milestone v1.18 (v1.17 code freeze)
- this is a regression introduced in 1.17, I actually believe we should fix that in 1.17.
/milestone v1.17 as noted above, this is a release blocking bug


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" "k8s.io/client-go/dynamic"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRD: Existing object: New object: says: If this is a problem with the new object definition rather than a bug with , I'd expect a better error message. (The new object looks fine to me, though; it should be identical to the old object.) What you expected to happen: I expected the object to be updated without an error. How to reproduce it (as minimally and precisely as possible): See description. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): CentOS Linux 7.4 (1708) Kernel (e.g. ): Linux master-head-1 4.13.4- SMP Wed Sep 27 13:32:23 EDT 2017 x8664 x8664 x8664 GNU/Linux Install tools: Custom scripts Others:
/sig api-machinery
Are you able to reproduce this everytime? I am able to use apply normally with your example.
I can reliably reproduce it if the annotation exists: If I use to remove the annotation, then it works the first time but fails thereafter:
This is happening for other CRDs as well (Prometheus Operator ServiceMonitors). If I apply a ServiceMonitor CRD 3 times: If I remove the annotation or replace the object () then I can work around the issue.
Seems any CRD is affected; the CRD used by our are also not working: This has completely broken our CI/CD workflow... unfortunately we'd already completed the TPR-CRD migration for the 1.8 upgrade before noticing this problem, so we can't really downgrade. How safe is as an alternative to ? It still generates an update event for watchers, not a delete/recreate, right?
It looks like something going wrong during the patching. After using changes from , I get this: /cc /area third-party-resource
But uses json merge patch and not strategic merge patch for custom resources:
I was able to reproduce this in v1.8.0 with the original manifests at the top. As found, it only happens on the 3rd execution: It also seems to only happen when the ought to be a no-op.

If I change something in the IPPool spec, works once more, then starts failing again. When the is a no-op, the PATCH request body generated by looks like this: When the is NOT a no-op, the PATCH request body looks like this:
There's a typo in the code generating the error messages for both and : They should be calling instead of . When I fix that, the error message becomes:
I can reproduce this without , just by doing a no-op twice:
The same issue affects v1.7.8 (tested on GKE)
Same issue with v1.8.1
This just fix the error message: But it doesn't fix the issue
Still investigating the issue but I noticed that when we do a no-op patch for native resources, we get a "not patched" response. However, for non-native resources, it still tries to patch it (and bumps the resourceVersion too).
I don't konwn whether this is a issue, I can't do strategic merge patch to CRD. When I run , --type=merge must be passed.
We should be able to apply strategic merge patch to CRDs. However, we can't apply strategic merge to CRs because we're missing the metadata about how to handle each field for the strategic merge patch. Unfortunately, this error is occuring even with . :(
/assign
do you know if the fix fir this will make it to 1.8.3
hopefully yes, this should be merged soon.
thanks! this has been a blocker for us to upgrade to 1.8
can you confirm this is completely fixed? The PR writes "partial fix".
Now we can't reproduce this issue with the method descripted above. I still need some I extra time to research it. Maybe we can close this one. If I can reproduce the issue in another way, I can open another issue with new situation.
If it's not fixed, let's keep it open.
Fun, a fork closing issues.
From Then I think we can officially close this.
(the fix is merged into release-1.8, but after v1.8.4-beta.0)
what stable release of k8s will this be in? 1.8.4 or 1.8.5?
Is there an update for this?

looks like it was merged here: but did not make it into 1.8.4, might make it into 1.8.5


Question: if message != "" { message = "last termination: " + message } <del> return "Running", translateTimestamp(state.Running.StartedAt), message, nil </del> <ins> stateMsg := "Running" if !status.Ready { stateMsg = stateMsg + " *not ready*" } return stateMsg, translateTimestamp(state.Running.StartedAt), message, nil </ins> } else if state.Termination != nil { return "Terminated", translateTimestamp(state.Termination.StartedAt), getTermMsg(state.Termination), nil }

When debugging , I found the output of 'kubectl get pods' was misleading, The status of the pod was reported as , but a further suggested the pod was not ready: shall warn the user about the fact that the pod is not ready.


Question: except zlib.error: # Don't fail if it's not gzipped. pass <del> data = data.decode('utf-8') </del> try: root = ET.fromstring(data)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --- ci-kubernetes-build-deprecated ci-kubernetes-build-deprecated-1-21 ci-kubernetes-build-1-20-deprecated ci-kubernetes-build-1-19-deprecated n/a - last good started at 2021-07-09 19:30:55 +0000 UTC - first bad started at 2021-07-09 21:14:51 +0000 TBD but likely related to the merge of or its followup PR's /sig testing /sig release /priority critical-urgent /area test /milestone v1.22 /assign
/triage accepted
Looks like maybe the needs to be moved to get picked up by whatever is invoking. Also, having a total lack of logs when bootstrap's default entrypoint fails is a fun failure mode. Ever more reason to stop using it (ref: )
should resolve, will hold open until we see successful job runs post-merge
/reopen I could have sworn I didn't use an auto-closing keyword...
Reopened this issue. <detailsIn response to : /reopen I could have sworn I didn't use an auto-closing keyword... Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close closing again, I see green runs for all of the affected builds:
Closing this issue. <detailsIn response to : /close closing again, I see green runs for all of the affected builds: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "bytes" "context" "encoding/base64" <ins> "flag" </ins> "fmt" "reflect" <ins> "regexp" </ins> "strconv" "strings" "testing" "time" <ins> genericapirequest "k8s.io/apiserver/pkg/endpoints/request" </ins> "k8s.io/apiserver/pkg/storage/value" aestransformer "k8s.io/apiserver/pkg/storage/value/encrypt/aes" kmstypes "k8s.io/apiserver/pkg/storage/value/encrypt/envelope/kmsv2/v2alpha1" "k8s.io/apiserver/pkg/storage/value/encrypt/envelope/metrics" "k8s.io/component-base/metrics/legacyregistry" "k8s.io/component-base/metrics/testutil" <ins> "k8s.io/klog/v2" </ins> kmsservice "k8s.io/kms/pkg/service" testingclock "k8s.io/utils/clock/testing" )

Add additional logging at level that associates UID with the metadata from the request. xref: /sig auth
/triage accepted
/milestone v1.26
Mind if I take a look at this? /assign
Hello I am wondering how's this coming along. Is there anything I can help with?
Sorry. I got a bit occupied with a different project.. Will have a PR opened for this for review tomorrow.. I am trying to find a way to inject the Verbs into log. But haven't found the right way to include that into the context.. Any suggestion on what can be done?
any suggestion on what is the best way to get the additional metadata for verbs into this log? Move that info across using ?
Hello , bug triage team here, just wanted to touch base with you for updates on this ticket. Thanks!
Thanks! This is being worked on and is still on track for .
There is a Pull request opened on this already as well. But Waiting on some clarification to see if that needs additional changes.
/reopen /assign /unassign
Reopened this issue. <detailsIn response to : /reopen /assign /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.27
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi Bug Triage Team for v1.27 here o/ Just wanted to touch base with you for updates on this ticket. Thanks!
I am not working on this anymore Carlos. This was re-assigned
This is tracked for . I've linked the PR.


Question: <ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package filters import ( "net/http" "runtime/debug" "github.com/golang/glog" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/util/runtime" apirequest "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/server/httplog" ) // WithPanicRecovery wraps an http Handler to recover and log panics. func WithPanicRecovery(handler http.Handler, requestContextMapper apirequest.RequestContextMapper) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { defer runtime.HandleCrash(func(err interface{}) { http.Error(w, "This request caused apisever to panic. Look in log for details.", http.StatusInternalServerError) glog.Errorf("APIServer panic'd on %v %v: %vn%sn", req.Method, req.RequestURI, err, debug.Stack()) }) logger := httplog.NewLogged(req, &w) var requestInfo *apirequest.RequestInfo ctx, ok := requestContextMapper.Get(req) if !ok { glog.Errorf("no context found for request, handler chain must be wrong") } else { requestInfo, ok = apirequest.RequestInfoFrom(ctx) if !ok { glog.Errorf("no RequestInfo found in context, handler chain must be wrong") } } if !ok || requestInfo.Verb != "proxy" { logger.StacktraceWhen( httplog.StatusIsNot( http.StatusOK, http.StatusCreated, http.StatusAccepted, http.StatusBadRequest, http.StatusMovedPermanently, http.StatusTemporaryRedirect, http.StatusConflict, http.StatusNotFound, http.StatusUnauthorized, http.StatusForbidden, http.StatusNotModified, apierrors.StatusUnprocessableEntity, http.StatusSwitchingProtocols, ), ) } defer logger.Log() // Dispatch to the internal handler handler.ServeHTTP(w, req) }) } </ins>

This is simple cleanup. It is unnecessarily disconcerting to read through apiserver logs and find every log line is from the file "". We should adjust the code printing these lines, either by changing the file name, or (ideally) by making it more intelligent about logging the source of the call (e.g., callers--).
, i'd like to try this one :).
No activity on this one, will try to fix it.
Has it been resolved? Can I try?
check out for discussion about it.


Question: addressObj.Address = existingIP } <del> address, err := gce.ReserveRegionAddress(addressObj, region) if err != nil { </del> <ins> if err = gce.ReserveRegionAddress(addressObj, region); err != nil { </ins> if !isHTTPErrorCode(err, http.StatusConflict) { return "", false, fmt.Errorf("error creating gce static IP address: %v", err) }

/kind bug What happened: Services of type LoadBalancer (GCE) with IP addresses promoted to static (by user or by orphan) cause the service controller to panic. The constant panicking will cause a reboot loop of the pod. How to reproduce it (as minimally and precisely as possible): a service on GCP it the GCP Console, promote the IP address of the LB to static and set the name to the loadBalancerName which is the same name as the forwarding rule. Environment: Kubernetes version: 1.7.0 AND 1.7.1 Cloud provider or hardware configuration: GCE Symptoms -controller-manager crash loops on the master. Mitigation** Option 1: Build your own version of branch for the master. Only the kube-controller-manager image needs to be updated. Option 2: For all affected services, set the field to the known external IP address. Steps: list of reserved IPs gcloud compute addresses list NAME REGION ADDRESS STATUS us-central1 35.35.35.35 INUSE list of all Services of type LoadBalancer kubectl get svc --all-namespaces | grep "LoadBalancer" default my-service LoadBalancer 10.0.99.201 35.35.35.35 80:/TCP,443:/TCP 58m each service of type LoadBalancer with an IP that's reserved (in the list above), continue to next step. the service's field equal to the known IP. Example_ apiVersion: v1 kind: Service metadata: creationTimestamp: 2017-07-13T00:09:31Z labels: app: echo name: my-service namespace: default resourceVersion: "" selfLink: /api/v1/namespaces/default/services/my-service uid: -675f-11e7-bee2- spec: clusterIP: 10.0.99.201 externalTrafficPolicy: Cluster loadBalancerIP: 35.35.35.35 <------- Field set to below. ports: name: port-1 nodePort: port: 80 protocol: TCP targetPort: 80 name: port-2 nodePort: port: 443 protocol: TCP targetPort: 443 selector: app: echo sessionAffinity: ClientIP type: LoadBalancer status: loadBalancer: ingress: ip: 35.35.35.35
/sig network /assign /area platform/gce
cc


Question: } validatePodReadiness := func(expectReady bool) { <del> err := wait.Poll(time.Second, wait.ForeverTestTimeout, func() (bool, error) { podReady := podClient.PodIsReady(podName) </del> <ins> err := wait.Poll(time.Second, time.Minute, func() (bool, error) { pod, err := podClient.Get(context.TODO(), podName, metav1.GetOptions{}) framework.ExpectNoError(err) podReady := podutils.IsPodReady(pod) </ins> res := expectReady == podReady if !res { <del> framework.Logf("Expect the Ready condition of pod %q to be %v, but got %v", podName, expectReady, podReady) </del> <ins> framework.Logf("Expect the Ready condition of pod %q to be %v, but got %v (pod status %#v)", podName, expectReady, podReady, pod.Status) </ins> } return res, nil })

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Since when has it been failing: It has been flaking more frequently since 3/13. Testgrid link: Reason for failure: The issue seems to be a timeout cc /kind flake /sig node /priority important-soon /milestone v1.14
Latest PR to touch that test May be due to a readiness probe failing.
/remove-kind failing-test sending this your way for triage too - could you have a look please? I wonder if it's a node problem or if timeouts are originating elsewhere fwiw, it's failing about 10%-15% of the time in e2e tests:
It looks like updating the status after the patch to the readiness gate takes longer than expected.

Looking at this failure: From the test logs: [It] should support pod readiness gates [NodeFeature:PodReadinessGate] ï¿½[1mSTEPï¿½[0m: submitting the pod to kubernetes ï¿½[1mSTEPï¿½[0m: patching pod status with condition "" to true ï¿½[1mSTEPï¿½[0m: patching pod status with condition "" to true Mar 14 01:52:56.720: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:57.731: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:58.725: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:59.707: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:00.721: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:01.708: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:02.753: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:03.707: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:04.713: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:05.709: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:05.767: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false From the kubelet logs: The pod goes to Ready at 1:53:12, and we stopped checking at 1:53:05 This looks expected, based on the description of the parameter in the test. I don't think this should be release-blocking.

/assign who owns this feature to determine how we can make the test less flaky. Feel free to close if this is already tracked somewhere else /unassign
Thanks, good to know. if it could be worked on, it would really help stabilize CI - which would be super appreciated, especially for release-blocking jobs.
When this tests was (2 or 3 release back), I remember the 10 second waiting timeout is enough for kubelet to react and update the pod ready condition. IIRC, kubelet reconciles pod status every 15s. But if pod readiness gate condition is updated, kubelet should reconcile ASAP skipping the 15s period. There might be a bug here.
Thanks - do you have time to take a look? It would help to know (especially if we need to include something in known issues).
I just did a few manual tests. Kubelet reacted very quickly. Not able to reproduce the update latency more than 10s.
I ran the test in a loop more than 50 times now on my own e2e cluster. Not a single failure. Is it because there are many tests running in parallel causing high load on kubelet, hence longer update latency?
Interesting, could well be the case Is there a way to validate that? And, is there something here we should raise in terms of performance/scalability (e.g. "update latency varies with pod load")?
That is a well-known result of the rate limiting of the kubelet. The kubelet can only send 5 qps of updates to the API Server, so the kubelet is limited in its ability to update statuses when there are lots of updates to send, which occurs when there is high pod churn.
This issue seems to not be a problem anymore. /close
Closing this issue. <detailsIn response to : This issue seems to not be a problem anymore. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Should this test be marked [Serial] then? (Or maybe flaky?) running into this again
Can you share the test grid link where it is flaky?
I've been testing extensively myself offline to re-evaluate tests to use in presubmit.

this sounds like inherently this test is not safe to run with lots of other tests in parallel, which would make it a candidate for
/reopen This is the flakiest test for PRs Triage link: Since this is a GA feature it should have conformance tests covering it, this specific test was prevented from promotion to conformance due to flakiness issues called out here One of the comments suggested rewriting to use an informer or poll for longer than 10s, and not use an explicit sleep. Tagging with [Serial] seems like it would work from a brute force perspective. Is the 10s number an important or documented SLO somewhere?
Reopened this issue. <detailsIn response to : /reopen This is the flakiest test for PRs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
another data point of sorts: IIRC I disabled this test in kind because it was too flaky, i wasn't aware that it was also flaky on GCE.
opened to deflake
Still seen after disabling auto flake retry:
top-ten flake according to
Confirming this flake is still top-ten after we merged Thanks for all your work here !
no longer top-ten, but still flaky -
back in the top ten according to
/assign assigning sig-node leads for visibility/routing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /remove-priority critical-urgent /priority important-longterm Not in the top ten, but it still flakes occasionally
I'm pretty sure we still blacklist this test in most kind CI because it's too flaky. Aaron Crickenberger < :
/milestone v1.19
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
Gonna push this with sig node /cc /remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale It's still failing across a number of jobs but I can't tell that it's as painful a flake today
/milestone clear
/assign /triage accepted
GitHub didn't allow me to assign the following users: fromanirh. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
working to reproduce outside CI - way easier to debug
still working my way towards a reproducer.
has been unable to reproduce locally. We are still seeing occasional timeouts in CI: Since this was the only timeout on that test run, we may want to consider tweaking the timeout thresholds?
I'm happy I have time again to work on this issue. It doesn't reproduce locally on BM, and this seems indeed to point to a too busy worker in CI and/or too short timeout. So let's start with the simplest fix:
Today in , it is the top 1 flake for "pr:pull-kubernetes-e2e-gce-ubuntu-containerd".
ok, time for me to prioritize this work
Do we have a testgrid link? I'm looking at to try to understand better the flakes and getting too little from here (but looks relatively good from here?)

More recent triage link: Appears to be failing in a number of containerd and crio jobs. /unassign /priority important-soon /remove-priority important-longterm
/milestone v1.22
Did we vet that this is fixed after ?


Question: }) <del> // This test verifies we don't allow scheduling of pods in a way that sum of local ephemeral storage limits of pods is greater than machines capacity. </del> <ins> // This test verifies we don't allow scheduling of pods in a way that sum of local ephemeral storage resource requests of pods is greater than machines capacity. </ins> // It assumes that cluster add-on pods stay stable and cannot be run in parallel with any other test that touches Nodes or Pods. // It is so because we need to have precise control on what's running in the cluster. ginkgo.It("validates local ephemeral storage resource limits of pods that are allowed to run [Feature:LocalStorageCapacityIsolation]", func() {

What happened: I am questioning validity of the test. Comments under of say: The test scenario described as it is sounds reasonable. The only nitpick I have is the very first sentence saying "". Pod's resource limits are ignored in predicates. They take effect in the scoring phase when priorities are computed. Also, priorities are computed only when there are at least two candidate nodes. So after revisiting the step number 5 it's impossible to make sure the additional pod does or does not get scheduled when setting the resource limits. There are either no candidate nodes or one. In which case the limits are ignored and the test passes in the first case and fails in the second. In case there are at least two candidate nodes, one of them will always get picked. In which case the additional pod gets scheduled (no matter what the resource limits field says). In the framework: resource limits are accessed through method [1] is used to perform pre-scoring [2] in the scoring phase score is computed through method [3] The test is alternating between passing and failing state. It passes when predicates give empty list of candidate nodes and failing when the list of candidate nodes is non-empty. Thus, making the pod resource limits ineffective. [1] [2] [3] What you expected to happen: Have another node schedule the additional pod validating that all saturated nodes (with scheduled) have lower score (0 in case of plugin). Or, invalidate the test by removing it. Can anyone check the test validity and confirm/disprove my statement about ineffectiveness of the resource limits? How to reproduce it (as minimally and precisely as possible): Make sure there is at least one node with non-zero free cpu resource before running the step 5. Anything else we need to know?: Environment: Kubernetes version (use ): master HEAD Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
cc
I have noticed you refactored the test in Can you still recall the original intent of the test? I can't grasp how the limits are actually taken into account.
I agree, I think the test is wrong, the scheduler doesn't make fit decisions on resource limits. In the test, should have set. do you mind fixing the test and update the test comment please?
On it /assign


Question: # Override this to use a different kubectl binary. kubectl=kubectl linux_deployment_timeout=60 <del> windows_deployment_timeout=300 </del> <ins> windows_deployment_timeout=600 </ins> output_file=/tmp/k8s-smoke-test.out function check_windows_nodes_are_ready {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add Support for Windows Server version 2004 and 20H2. Why is this needed: Windows Server version 1909 is going end of life.
/sig windows /triage accepted


Question: description: Create a StatefulSet resource. Newly created StatefulSet resource MUST have a scale of one. Bring the scale of the StatefulSet resource up to two. StatefulSet scale MUST be at two replicas. <del> release: v1.16 </del> <ins> release: v1.16, v1.21 </ins> file: test/e2e/apps/statefulset.go - testname: StatefulSet, Rolling Update with Partition codename: '[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic]

[ ] two weeks soak start date : testgrid-link [ ] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining StatefulsetScale endpoints that are untested. with this query, you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested, stable, apps endpoints sql-mode select distinct endpoint, right(useragent,73) AS useragent from testing.auditevent -- where useragent ilike '%subresource%' where endpoint ilike '%AppsV1NamespacedStatefulSetScale%' and releasedate::BIGINT round(((EXTRACT(EPOCH FROM NOW()))::numeric)1000,0) - and useragent like 'e2e%' order by endpoint limit 30; example endpoint | useragent -----------------------------------------|--------------------------------------------------------------------------- patchAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] readAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] replaceAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] (3 rows) If a test with these calls gets merged, test coverage will go up by 1 point* This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: _, tag, _, _ := parsers.ParseImageName(obj.Image) // Check image tag if tag == "latest" { obj.ImagePullPolicy = PullAlways } else {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): initContainers imagePullPolicy initContainers required value Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: cluster/kube- Others: What happened: When trying to create either a ReplicaSet or Replication Controller whose Pod template contains an Init Container, the validation fails saying is a required value. What you expected to happen: The create should succeed and the Init Container should receive a default . How to reproduce it (as minimally and precisely as possible): Try to create either of the following: ReplicaSet Replication Controller Anything else do we need to know: If I add to the Init Container in the ReplicaSet or Replication Controller Pod template, the create succeeds. When I create an Init Container without in a StatefulSet Pod template, it succeeds. When I create an Init Container without in a bare Pod, it succeeds:
Looks like defaulting not being applied on server? That is probably due to removing defaulting from convert. Surprised we didn't catch this earlier
Some values still not being defaulted
, I'm going to create a PR if none works on it.
We decode from bytes, set defaults and then convert; but for init containers, we delay the decode from "decode phase" to "convert phase" at .

The defaults has been handled for initContainer at . /cc I'm working to find a solution for that; if any suggestion, please let me know.
Would this bug affect the deletion of jobs with an init-container? I ran into this bug and adjusted my manifest to add the pull policy to the init-container. However, I now cannot delete said job claiming a field is immutable. Example:
, not sure now. Let me check it,
Same error here with version 1.5.1, 1.5.2 The Job "grafana-import-dashboards" is invalid: [0].imagePullPolicy: Required value With this setup: kubectl apply -f monitoring Not working even if I specify a pullpolicy in the yaml. If I roll back to 1.4.7, it is working. Thank you. Greg.
any updates on a PR to address this issue? Is anyone from able to pick this issue up otherwise?
, here's the latest status: there's is a PR to generate for each Object (), it works for Object (Pod, Job) now but List; I'm working to make it generating for List.
Why would you need List to work? No client side logic should be decoding lists except as a generic submission.
The List tests are failed in . There's no for List, so no code to decode init-container annotation, e.g. of should call 's recursively.
Thanks for the update
+1
Any update on this? I still getting this issue with the last version of 1.5.3 on GKE?
ping for the comments? should we merge the PR for 1.5?
, there's a PR () that handled that issue; but we're discussing not to merge it because initContainers had been moved to fields, and fields is the recommended way to use since 1.6. But I'd like to get & 's comments on 1.5.
This is still an issue in 1.5.3 when using annotations: Without the deployment would not be accepted
ping & :).
i dont think this is fixed in 1.5 I just encountered this bug on 1.5.3
, yes, the PR is NOT merged to any branch, so the issue is still here. I'll create a PR for 1.5.
Hi Anyone help this issue.

#cat mongo- apiVersion: v1 kind: Pod metadata: labels: name: mongo-svc-a name: mongo-deployment spec: replicas: 1 template: metadata: labels: name: mongo spec: containers: image: mongo name: mongo ports: name: mongo containerPort: hostPort: volumeMounts: mountPath: "/data/db" name: mongopv volumes: name: mongopv persistentVolumeClaim: claimName: mongopvc When i want create mongo db container by using this file folowing error is occured. The Pod "mongo-deployment" is invalid:spec.containers: Required value
, it seems there're two here, please remove the first one and have a try.
Hi my kubectl binary is configured at ubuntu 16.04 machine.

NAME LABELS STATUS 10.1 Ready,SchedulingDisabled 10.2 10.1.10.2 Ready 10.3 Ready apiVersion: v1 kind: Node metadata: creationTimestamp: 2017-03-03T14:41:52Z labels: 10.1.10.1 name: 10.1.10.1 resourceVersion: "" selfLink: /api/v1/nodes/10.1.10.1 uid: -001f-11e7-8e83- spec: externalID: 10.1.10.1 status: addresses: address: 10.1.10.1 type: LegacyHostIP address: 10.1.10.1 type: InternalIP capacity: cpu: "2" memory: 4041352Ki pods: "110" conditions: lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-03-03T14:41:52Z message: kubelet has sufficient disk space available reason: KubeletHasSufficientDisk status: "False" type: OutOfDisk lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-04-05T15:27:52Z message: kubelet is posting ready status reason: KubeletReady status: "True" type: Ready nodeInfo: bootID: -6124-4ac7-ab96- containerRuntimeVersion: docker://1.12.6 kernelVersion: 4.9.16-coreos-r1 kubeProxyVersion: v1.2.4+coreos.1 kubeletVersion: v1.2.4+coreos.1 machineID: osImage: Container Linux by CoreOS 1298.7.0 (Ladybug) systemUUID: C3FD9852-2DFB-21F4-CE07-2597A746D99D When i create pv for creation pod is shows pv is pending. Please give any suggestions.....
Is there solution of this problem with version 2.7. Still facing the same. what exactly expectation of helm upgrade --install relname chartsdirectory is is expected to work on first time installation in case with CI/CD Pipelines.


Question: awsTags = append(awsTags, tag) } <ins> backoff := wait.Backoff{ Duration: createTagInitialDelay, Factor:   createTagFactor, Steps:    createTagSteps, } </ins> request := &ec2.CreateTagsInput{} request.Resources = []*string{&resourceID} request.Tags = awsTags <del> // TODO: We really should do exponential backoff here attempt := 0 maxAttempts := 60 for { </del> <ins> var lastErr error err := wait.ExponentialBackoff(backoff, func() (bool, error) { </ins> _, err := c.ec2.CreateTags(request) if err == nil { <del> return nil </del> <ins> return true, nil </ins> } // We could check that the error is retryable, but the error code changes based on what we are tagging // SecurityGroup: InvalidGroup.NotFound <del> attempt++ if attempt > maxAttempts { glog.Warningf("Failed to create tags (too many attempts): %v", err) return err } </del> glog.V(2).Infof("Failed to create tags; will retry.  Error was %v", err) <del> time.Sleep(1 * time.Second) </del> <ins> lastErr = err return false, nil }) if err == wait.ErrWaitTimeout { // return real CreateTags error instead of timeout err = lastErr </ins> } <ins> return err </ins> } // Finds the value for a given tag.

In a cloud provider environment it's possible to use up the API quota grinding the cluster to a halt: In a multi tenant environment, one user could consume the quota for an entire cluster. Some cloud providers allow throttling of API calls by type but Kube does not differentiate on the call types, so DOS still possible.
Adding some colors. Cloud providers like AWS/GCE/Azure have rate limits on API calls. GCE and Azure have fixed API calls per second, while AWS doesn’t offer fixed quota. Once API call quota are exhausted, no more API calls are allowed. Consequences include: If the error code path are not well tested, unexpected behavior will happen. A malicious user may launch DoS by triggering frequent API calls. The entire cluster would be impacted AWS cloudprovider has implemented API throttling at global level as well as on per call type level most recently. However, at least on AWS, quota for call types are different: query type API calls have higher quota while modification type API calls have lower quota. There should be strategies to differentiate these calls. Quota for Cloudprovider API calls should be supported on per account/project basis, so DoS can be averted. (Per This needs significant enhancements in Attach/Detach logic, volume plugin gets only pv.Spec, which does not include information about namespace of the PVC bound to it nor the user that initiated the operation. And only volume plugin (on top of cloud provider) can throttle API calls.) Throttling at per call type level should be supported.
I understand there is a problem, but I am not convinced on any particular solution. Can we get a more detailed enumeration on the code paths that invoke cloud provider backed interfaces, and where we would want some bucket to control rate for those calls? I don't think the existing object should bleed concepts from the underlying cloud provider, and I don't think its setup to support rate based quota models at this time.
/cc
I went through storage parts of the cloud provider and on AWS it looks like this: Deleter (in persistent volume controller) calls ec2.DeleteVolume() once per delete attempt Provisioner (in persistent volume controller) calls ec2.DescribeInstances() with filter "instance-state-name" == "running" to get all zones that contain a node. Once per provisioning attempt.

calls ec2.CreateVolume() once per provisioning attempt calls ec2.CreateTags() - up to 60 times per provisioning call (until it succeeds). Typically one attempt needed. calls ec2.DescribeVolumes() once per provisioning attempt When provisioning or deletion of a volume fails, the controller re-tries every 15 seconds until it succeeds. Typically it succeeds on the first attempt. PersistentVolume admission plugin: ec2.DescribeVolumes() once per new PV instance created (to fill its labels) Attach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attach attempt (to get list of already attached volumes) ec2.AttachVolume() once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's attached Detach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attempt ec2.DetachVolume once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's detached please confirm how often the controller re-tries on failure. Is it ? With many volumes being attached/detached at the same time, each volume polls its own DescribeVolumes() every 10 seconds while waiting for the operation to finish. We could save a lot of calls if we had one DescribeVolumes() call for all "interesting" volumes and distribute the results to proper places. In addition, some exponential backoff would be helpful, AWS is pretty slow when attaching/detaching volumes. Only storage was investigated!!!
The same issue also applies to ELB on AWS. Service controller invokes which calls:
It should be noted exponential backoff is disabled in - , I am not sure if it was intentional. Also, we should consider case where k8s cluster is in, inconsistent state. For example - if volume being deleted is still mounted to a pod, there would be a flood of API requests for deleting volume in pvcontroller (without any exponential backoff)
You're right, PV controller does not use retry logic of GoRoutineMap and retries provisioning and deletion on its own and currently without exponential backoff. The reason is that PV controller can react to deletion of object, e.g. when user removes a PVC then the controller does not need to retry provisioning a volume for it.

PV controller retries failed operation every 15 seconds by default. Some exponential backoff could be if 15 seconds are too short.
If Attach or Detach operation fails (e.g. due to throttling), there already is exponential backoff, starting with 0.5 seconds and doubling with each failed attempt, capped at 2 minutes. Summary: there already is exponential backoff for Attach/DetachVolume with we will have the same exponential backoff for Create/DeleteVolume the only fixed is DescribeVolumes() when we wait for a volume to get really attached/detached. I am going to fix it.


Question: WriteConfigTo string // CleanupAndExit, when true, makes the proxy server clean up iptables and ipvs rules, then exit. CleanupAndExit bool <del> // CleanupIPVS, when true, makes the proxy server clean up ipvs rules before running. CleanupIPVS bool </del> // WindowsService should be set to true if kube-proxy is running as a service on Windows. // Its corresponding flag only gets registered in Windows builds WindowsService bool

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-proxy don’t clean ipvs rules with . What you expected to happen: kube-proxy clean ipvs rules with by default. How to reproduce it (as minimally and precisely as possible): Run kube-proxy in ipvs mode, and clean rules with . Anything else we need to know?: Based on , should cleanup all rules by default, including ipvs rules. And also retains functionality. Environment: Kubernetes version (use ): v1.20.0-bate.0 Cloud provider or hardware configuration: OS (e.g: ): centos7 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network
it seems that this flag entered deprecation, so I think that recovering functionaly is not the right approach you were involved in the discussion with this flag in , can you take a look at this?
thanks for your reply. The was in , to avoid removing IPVS rules which are not created by kube-proxy. According to , It seems that deleting ipvs rules by default is expected, but it is not so. I think that keeping --cleanup-ipvs is meaningful, unless kube-proxy can only delete the IPVS rules created by itself.
/assign
sorr for the late answer , but the flag must have the same functionality than , actually I can see that it has implemented. In case that the is not working we should fix it, but not recover the deprecated flag
Agree
sorry if I'm missing something. I just try to make the function of as described, not to recover the flag. The problem I see is: The can't flush IPVS. Because the IPVS flush process is still controlled by . Beforce , it was assigned by flag, but now it is not assigned, always false. currently does not have any functionality, just stored by . So, should we keep the function of temporarily by assigning it to ,or just make always flushing IPVS regardless of the ?

I prefer the former because the flag is not completely abandoned :)
it seems that is what the flag description says, you have to use to flush the ipvs entries if that is the case it seems that we are ok, right? no changes needed
In fact, can't flush ipvs entries. The reason is what I described above.
I´m sorry, I think I got it know, the flag is not exposed, but the option is there and initialized to true, this is what I see so, it seems that flush IPVS by default
yes, should flush IPVS by default, but it doesn’t actually
... and finally I got it, so sorry :bow: the ProxyServer is not being instantiated with the flag, despite is set :facepalm: you are more familiar with the IPVS implementation than me, I personally think that, if the flag deprecation has started, we should make that always flush IPVS and totally remove ... it seems aligned with the other proxy implementations
/triage accepted
Actually, after doing some code archaeology, this was the expectation per And the flag clearly mentions that --cleanup must flush IPVS entries too I think that we can go ahead and remove the --cleanup-ipvs flag and make --cleanup to flush ipvs too, right now is a mess and is not working at all , are you going to submit a PR?
thanks for digging up the history on this. That lines up with my recollection and the KEP around kube-proxy's clean up logic
OK, I will create a PR to remove the flag


Question: ) type pullResult struct { <del> imageRef string err      error </del> <ins> imageRef     string err          error pullDuration time.Duration </ins> } type imagePuller interface {

While going over Kubernetes events I noticed that image pulls for some small images were taking several minutes according to the "Successfully pulled image" event. From investigating this is misleading as the actual pull time is much smaller but due to --serialize-image-pulls=true being the default in the observed cluster and other pods being initialized at the time, the reported pull time ends up accounting for time while waiting for other pods to finish pulling. I expected to see the time it takes to download the image itself from a registry. Customers seem to rely on this metric for this information and we have received some complaints as a result where customers noted slow pull times as a result of misunderstanding the current behavior of this metric. Start with a K8s cluster with --serialize-image-pulls=true (default) - For my check I used default minikube v1.26.1 pod-large where pod-large pulls any large image (the image cannot be locally cached) after step 1 (within seconds) create pod-small where pod-small pulls any small image Check the output for the event after running: Note that despite the small size, pull time for the image in pod-small is several minutes, we should instead see it take seconds (which is how long the pull actually takes). To note this is the case you can try pulling another small image like alpine that isnt cached locally and compare after the above are done. Here is the code that pulls an image in kubernetes and logs the event: As can be seen from it, it's not true pull time, just time from invoking the puller to receiving a result. The serial puller is using a Golang channel as a queue and will just wait until a previous image is done pulling as I noted before, thus it will end up recording the amount of time in the queue + pull time yielding a misleading metric. <details</details<detailsShould reproduce anywhere (tried it on minikube) but have personally observed this in Azure as well </details<details</details<details</details<details</details<details</details
/sig node
But means that kubelet can pull multi images at the same time. For kubelet, is by default true. is for promoting the use of parallel image pulls. The problem maybe with . I did some simple test with 3 ubuntu images as below. The time was collected from events.

-- --serialize-image-pulls=true -- -- image pulling serialize ubuntu:22.10 73.6 MiB 1m21.284220239s ubuntu:20.04 75.7 MiB 2m7.876426944s ubuntu:14.04 212.9 MiB 4m52.884660938s
You are right thanks for taking a look at this. It seems I got the two reversed in my head. The issue indeed happens when --serialize-image-pulls=true which is the default as you pointed out. I'll update the issue to reflect that
/triage accepted /priority important-longterm


Question: } func expectResult(t *testing.T, w *worker, expectedResult results.Result, msg string) { <del> result, ok := resultsManager(w.probeManager, w.probeType).Get(testContainerID) </del> <ins> result, ok := resultsManager(w.probeManager, w.probeType).Get(w.containerID) </ins> if !ok { t.Errorf("[%s - %s] Expected result to be set, but was not set", w.probeType, msg) } else if result != expectedResult {

Happened in GKE suite:
Does it need a liveness probe, maybe?
to delegate
Saw
Can you take a look at this one? Thanks!
FWIW, I saw some fishy messages in in (node: 104.197.48.84) It seems like docker was giving conflicting view of the container status, i.e., whether the container was running or not (and whether kubelet should kill it). kubelet's first attempt to kill the container : All the exec well after the grace period still shows that the container is the "Running: true" state () Later, kubelet tried killing the container again and failed because container does not exist. Note that after the failed kill also shows that the container was in the Running state. At the same time, cadvisor observed the container being stopped but then detected the container and then tried adding it again, but failed to find the /cc UPDATE: According to supervisord, docker was not killed during this time.
I didn't find result for the first Killing, does that really return? What is the log level, I remember that you told me before the log level in GKE is 2?
AFAICT, the pod worker may have never returned because the specific pod worker didn't log anything beyond that point. I was looking at (docker stop hangs) to see if it's relevant. didn't print any error though.
Yeah, the last SyncPod log was which is before the first Killing. Then the pod worker was stuck in
This really looks similar with I tried to reproduce it following the comment. The container was not actually killed. Docker stop hangs. Docker ps shows the container is still running. But the difference is that when I tried to kill it with another , it still hung there.
I was able to reproduce this quite easily by running 30 pods with the same spec in the test and with :
From
My theory is that during may cause docker to leave the container in a unrecoverable state (yep, a bug). I can consistently reproduce this by simply creating (which always fails), and leave the cluster for < 10 min. The restart counts of some of the pods will stop increasing after sometime, and after ssh'ing in to the node, one can easily confirm the same symptoms posted above. If pods use the http probe, they do not have this issue.

For the workaround, we can instruct the probe workers to stop probing after detecting a liveness check failure, until the container has been restarted (with a new container ID). Note that this is just a workaround, as it doesn't stop other components or users to run during this window. I've verified that with this fix, there are no zombie docker containers anymore. Will upload a PR.
FWIW, I also tested it with docker 1.8.3, which has the same problem.
Thanks !


Question: Properties: map[string]spec.Schema{ "maxUnavailable": { SchemaProps: spec.SchemaProps{ <del> Description: "The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding up. This can not be 0 if MaxSurge is 0. By default, a fixed value of 1 is used. Example: when this is set to 30%, the old RC can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old RC can be scaled down further, followed by scaling up the new RC, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods.", </del> <ins> Description: "The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding down. This can not be 0 if MaxSurge is 0. By default, a fixed value of 1 is used. Example: when this is set to 30%, the old RC can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old RC can be scaled down further, followed by scaling up the new RC, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods.", </ins> Ref:         spec.MustCreateRef("#/definitions/intstr.IntOrString"), }, },

API documentation says is rounded up, but it is actually rounded down: The code shows it is rounded down, not up: and
Yes, the documentation is obsolete. We changed maxUnavailable to round down so that we wouldn't accidentally infer 100% unavailability for non-100 percentages. For example, maxSurge=25% and maxUanavailable=25% with a Deployment of 1 replica meant that we could scale down 1 before thus leading in 100% maxUnavailable (implicit).