Question: e2epod.DeletePodWithWait(f.ClientSet, pod) }() <del> err := e2epod.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PodStartShortTimeout) framework.ExpectError(err, "while waiting for pod to be running") </del> <ins> err := e2epod.WaitForPodContainerToFail(f.ClientSet, pod.Namespace, pod.Name, 0, "CreateContainerConfigError", framework.PodStartShortTimeout) framework.ExpectNoError(err, "while waiting for the pod container to fail") </ins> } func newPod(command []string, envVars []v1.EnvVar, mounts []v1.VolumeMount, volumes []v1.Volume) *v1.Pod {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: N/A Which test(s) are failing: should fail substituting values in a volume subpath with absolute path Since when has it been failing: N/A Testgrid link: N/A Reason for failure: Some of the tests are negative test cases which are supposed to ensure that those invalid usecases are handled properly. However, some of the tests are false positives, they can pass due to various reasons. One such example is: should fail substituting values in a volume subpath with absolute path. This test can pass if: the Pod cannot start due to various reasons (e.g.: the container image cannot be pulled or does not exist). the Pod ran to completion, even though the container was not supposed to start in the first place. Anything else we need to know: /sig testing Related:
/sig storage SIG storage owns the test case, though I see that you are changing core framework methods to fix this, so leaving SIG testing. Looks like also possibly sig-node, skimming the linked PRs Based on the PRs this also appears to be related to windows? (unsure, no mention here)
I've Related: to the issue description. After fixing this test, I've noticed that it now fails on Windows, which means the issue was masked by the test's false positive result. Basically, it means that you can have volume subpaths with absolute paths substituted on variable expansion and have Windows containers use them and be running, which according to the test, it shouldn't be a supported usecase. The fix should probably be also backported to previous releases.


Question: le.config.Lock.RecordEvent("stopped leading") glog.Infof("failed to renew lease %v: %v", desc, err) cancel() <del> }, 0, ctx.Done()) </del> <ins> }, le.config.RetryPeriod, ctx.Done()) </ins> } // tryAcquireOrRenew tries to acquire a leader lease if it is not already acquired,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After merged, the leader election is calling too many api calls. Previously it calls with a 2s interval by default. Now it calls with 0 interval. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
/assign
/kind bug


Question: return nil } <del> switch mode := info.Mode(); { case mode.IsDir(): </del> <ins> mode := info.Mode() if mode.IsDir() { </ins> if err := w.fsWatcher.Add(path); err != nil { return fmt.Errorf("failed to watch %s, err: %v", path, err) } <del> case mode&os.ModeSocket != 0: </del> <ins> } else if isSocket, _ := util.IsUnixDomainSocket(path); isSocket { </ins> event := fsnotify.Event{ Name: path, Op:   fsnotify.Create,

Looking at [1], we can see that the plugin Watcher is traversing the plugin directory, and check whether a new file is a unix socket or not. However, : Those file should not have been ignored. Instead of using, which also has a Windows implementation [2]. This issue may also cause the unit tests to flake [3]. [1] [2] [3] Instead of using, which also has a Windows implementation [2]. This will result in the plugin Watcher to work as intended. Run the ` unit test. No response Kubernetes v1.25, v1.24. N/A Windows Server 2019 or newer. <details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows /sig testing /milestone v1.25 /priority important-soon
/assign


Question: tag.Value = aws.String(TestClusterId) tags := []*ec2.Tag{&tag} <del> var runningInstance ec2.Instance runningInstance.InstanceId = aws.String("i-running") runningInstance.PrivateDnsName = aws.String(string(nodeName)) runningInstance.State = &ec2.InstanceState{Code: aws.Int64(16), Name: aws.String("running")} runningInstance.Tags = tags var terminatedInstance ec2.Instance terminatedInstance.InstanceId = aws.String("i-terminated") terminatedInstance.PrivateDnsName = aws.String(string(nodeName)) terminatedInstance.State = &ec2.InstanceState{Code: aws.Int64(48), Name: aws.String("terminated")} terminatedInstance.Tags = tags </del> <ins> var testInstance ec2.Instance testInstance.PrivateDnsName = aws.String(string(nodeName)) testInstance.Tags = tags </ins> <del> instances := []*ec2.Instance{&terminatedInstance, &runningInstance} awsServices.instances = append(awsServices.instances, instances...) </del> <ins> awsDefaultInstances := awsServices.instances for _, awsState := range awsStates { id := "i-" + awsState.state testInstance.InstanceId = aws.String(id) testInstance.State = &ec2.InstanceState{Code: aws.Int64(awsState.id), Name: aws.String(awsState.state)} </ins> <del> c, err := newAWSCloud(CloudConfig{}, awsServices) if err != nil { t.Errorf("Error building aws cloud: %v", err) return } </del> <ins> awsServices.instances = append(awsDefaultInstances, &testInstance) </ins> <del> instance, err := c.findInstanceByNodeName(nodeName) </del> <ins> c, err := newAWSCloud(CloudConfig{}, awsServices) if err != nil { t.Errorf("Error building aws cloud: %v", err) return } </ins> <del> if err != nil { t.Errorf("Failed to find instance: %v", err) return } </del> <ins> resultInstance, err := c.findInstanceByNodeName(nodeName) </ins> <del> if *instance.InstanceId != "i-running" { t.Errorf("Expected running instance but got %v", *instance.InstanceId) </del> <ins> if awsState.expected { if err != nil || resultInstance == nil { t.Errorf("Expected to find instance %v", *testInstance.InstanceId) return } if *resultInstance.InstanceId != *testInstance.InstanceId { t.Errorf("Wrong instance returned by findInstanceByNodeName() expected: %v, actual: %v", *testInstance.InstanceId, *resultInstance.InstanceId) return } } else { if err == nil && resultInstance != nil { t.Errorf("Did not expect to find instance %v", *resultInstance.InstanceId) return } } </ins> } }

If a node is attached to a cloud provider and the machine is in any state other than running/available/ready/whatever cloud provider uses to indicate ready, it will be removed from the kubernetes cluster. Looking at the various cloud providers, some of them filter nodes based on the state whereas others do not. The node controller has logic for handling a node that is not updating status and evicting pods/tainting the node as needed. Is there a reason why the cloud providers only return instances/vms that are "ready" instead of returning any instance/vm that matches the requested name? It seems like it might be more consistent to have the cloud providers only filter on name and allow the caller to determine what to do if the found instance isn't in a desired state.
If that's really the case it's unfortunate - we probably can fix logic to list everything (I guess there's a flag that you can pass to CP to give you everything, not only working machines). I have no idea who deals with that ? OTOH I think the only problem with this would be if the machine would flap, as this would be annoying and cluster autoscaler might get crazy ( If it's really unready/unavailable/whatever then it's fair to assume that nothing's working on it (if StatefulSet can live with that assumption
do you think the NodeController should/should not delete a Node from storage in this situation: Node is configured to use a cloud provider, VM is powered off but not deleted? I would argue for "should not delete".
A trivial answer is that PrivateDNSNames are not unique on AWS. A running instance and a stopped instance could have the same IP, hence same PrivateDNSName. Rob / Marek: Can you clarify with a scenario where something unexpected happens: what that is and what you expect to happen?
can we work around that somehow for ec2?
I don't know what we are working around! The concern is AWS instances that are Stopped but not Terminated? Separately, we've been working since before 1.0 to get us off PrivateDNSName on AWS though...
the issue is that, depending on cloud provider, the NodeController may delete a Node from etcd if the VM is not Running.

If you power off a VM, wait longer than the NodeController's eviction threshold (5 minutes by default), and then power the VM back on later, the NodeController will have deleted the Node from etcd in the interim. Some cloud providers in the kube code base expect an instance to be Running, and if it's not, they don't include that instance in the list of instances, or a "get instance" call returns "not found" (I'm not sure which, has the full details), and so the NodeController doesn't see the powered off VM and deletes the Node.
That is the current behaviour, but you've skipped the bit where something goes wrong :-) Even if we fix this problem though do we close this issue? I would vote to close this issue now on the basis that there's no objective measure for when we can mark this issue resolved, and that we start again with a problem statement like the one you're describing, which feels actionable.
the intent of this issue was what I described. It's possible that it wasn't clear in the initial description.
At least on AWS: The motivation for the filtering is that Terminated instances still have the old PrivateDNSName listed (IIRC), and so the names aren't unique unless you exclude Terminated. But we don't have to exclude Stopped instances (power-off), and the nodes will just get marked NotReady. I don't think that will cause problems, but it may be that the correct place to fix is actually somewhere else - e.g. on kubelet rejoin. What goes wrong when the node tries to rejoin?
At least 1 thing: if you have manually labels to the node post-registration, those labels are gone when it rejoins.
That sounds like it would be correct for some classes of labels.
- small clarification: 5 min threshold has nothing to do here. It only says that if Node is unresponsive for 5 or more minutes we'll start evicting Pods from that Node (modulo rate limiting). Deleting non-existing Nodes is a separate thing - when Node skips few heartbeats (currently 4) NC checks if cloud provider still returns the instance corresponding to that Node, and if not then delete the Node instance. IIUC what wrote the current behavior is valid for AWS, as Node won't come back (it's DNS name might be used by something else).

This isn't the case for only aws. Looking at the code it appears at least openstack, aws, vsphere, ovirt, rackspace cloud providers also filter on available node. I wasn't able to verify whether azure or others were doing the filtering. If they weren't, then at best we have inconsistency with different cloud providers. IMO we shouldn't be deleting a node from kubernetes if it is just stopped or some state other than available. If the instance exists in the cloud provider then imo it should exist in kubernetes, but possibly marked unscheduable if needed.
ouch, that's even worse! Isn't that only 40-50 seconds? and I chatted yesterday and I think there are a couple of possible ways to look at this: 'm used to doing things the "old way" where I treat my servers as pets, and I want to be the one who decides when to retire/delete a server. 'm willing to treat servers as cattle, and if I ever need to take a server down so I can upgrade the kernel/packages/etc, or if a server goes offline for whatever reason, I'll just throw it away and replace it with a newer one. I'd be willing to bet that most IT organizations are still operating under 1). Justin recommended that we need to do a better job at education, and while I agree with that, there is still going to be a transitional period where sysadmins want to be making these kinds of decisions, and they're not yet willing to let the cluster automatically delete nodes. In summary, I think at the very least we need to: this behavior, if it's not already documented an option to the node controller to disable this behavior (and possibly disable it by default) more on evangelizing how we envision server management to happen in a Kubernetes world
Yes, I believe it's 40s. I'm curious why we don't want to just change the implementation of cloud provider, so that we'll use 'full list' (i.e. one including not running Nodes, so only delete ones that are really not present) in NC?
That also works for me, assuming it doesn't cause any issues.
It's strictly better than disabling it altogether, right?

The drawback would be that if you create a VM which will get the same name (I guess it's a DNS name), than the suspended/whatever one, then it'll be treated by k8s as the old one, which may cause a issues when running StatefulSets (especially if old machine didn't release PV or sth, as IIUC k8s will believe that it's attached to the new machine But if you're willing to do that, then I don't see anything that will prevent you from doing so, assuming it will be flag gated.
I think stateful sets are unaffected by this unless both new and old machines are booted at the same time AND either kubelet fails to terminate the pods. I can't think of how that would happen. Not being able to unmount a volume could be caused by other admin actions and is something we can warn about.
For a change of this significance, we'll want an e2e showing the problem, and then a PR that lets the e2e pass.
I'm not sure an e2e is the best way to show this. An e2e would incur costs to run on a cloud provider so likely wouldn't be part of any standard e2e run. I think a unit test could be done though.
The PR I pushed only changes AWS. w/o this change the node is removed from kubernetes shortly after it is no longer running. With this PR the node transitions to unschedulable and details show it hasn't reported its health check.
PTAL and let me know if you see issues with this approach
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
/remove-lifecycle stale


Question: }) /* <del> Release: v1.17 </del> <ins> Release: v1.17, v1.21 </ins> Testname: aggregator-supports-the-sample-apiserver Description: Ensure that the sample-apiserver code from 1.17 and compiled against 1.17 will work on the current Aggregator/API-Server.

[x] test pr : kuberenetes/kubernetes# [x] two weeks soak start date : testgrid-link [x] two weeks soak end date : [x] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. With this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Addressing the following endpoints in this document; Using the current e2e test as a template, create an APIService for Using the REST client, read the APIService status (readApiregistrationV1APIServiceStatus) and confirm the value of . Using the REST client, patch the APIService (patchApiregistrationV1APIService) before confirming that it has been updated. Using the REST client, list all APIServices in the cluster (listApiregistrationV1APIService) before confirming that can be found. Clean up APIService resources used by the test. Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. This query shows the endpoints hit within a short period of running the e2e test example endpoint | useragent ---------------------------------------|
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator2 (6 rows) Based on the initial APIsnoop query and the results listed above there are three new endpoints hit.

If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted


Question: package metrics import ( <del> "os" </del> "time" <del> "github.com/prometheus/procfs" </del> "k8s.io/klog/v2" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Start kubelet on a Windows node and you will get this error: What you expected to happen: The process start time to be read appropriately for Windows How to reproduce it (as minimally and precisely as possible): Start kubelet on a Windows node Anything else we need to know?: The gauge falls back to but we get this error. This information is avaliable via the windows package so we should be able to get an accurate time. Environment: Kubernetes version (use ): 1.20.1 Cloud provider or hardware configuration: azure OS (e.g: ): Windows 2019 Kernel (e.g. ): Version 10.0..1637 Install tools: aks-engine (but would be any) Network plugin and version (if this is a network-related bug): Others: /sig windows /assign
/triage accepted


Question: // decodeDockerConfigFieldAuth deserializes the "auth" field from dockercfg into a // username and a password. The format of the auth field is base64(<username>:<password>). func decodeDockerConfigFieldAuth(field string) (username, password string, err error) { <del> decoded, err := base64.StdEncoding.DecodeString(field) </del> <ins> var decoded []byte // StdEncoding can only decode padded string // RawStdEncoding can only decode unpadded string // a string is correctly padded if and only if its length is a multiple of 4 if (len(field) % 4) == 0 { // decode padded data decoded, err = base64.StdEncoding.DecodeString(field) } else { // decode unpadded data decoded, err = base64.RawStdEncoding.DecodeString(field) } </ins> if err != nil { return }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: docker now provides a new favorite way to store the credentials with the variable credsStore in the file it it is set to desktop, which will be using the executable docker-credential-desktop whose source code is docker-credential-desktop doesn't pad the auth field. So, if we use the command k8s credential provider won't be able to decode the field because of the error Failed to pull image "myrepo/myimage:tag": illegal base64 data at input byte What you expected to happen: Image pulled normally How to reproduce it (as minimally and precisely as possible): open docker config file, and have credsStore set to desktop. Run docker login and then use the to create a k8s dockerconfigjson secret, the error will directly show Anything else we need to know?: I wrote a patch that I will submit Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-19T12:38:36Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64" Cloud provider or hardware configuration: minikube version: v1.3.1 commit: OS (e.g: ): Mac Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig Apps
/remove-sig apps /sig auth /sig node cc to triage


Question: ], importpath = "k8s.io/kubernetes/test/integration/util", deps = [ <ins> "//pkg/api/legacyscheme:go_default_library", </ins> "//pkg/cloudprovider/providers/gce:go_default_library", "//pkg/scheduler:go_default_library", "//pkg/scheduler/api:go_default_library",

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: I0116 21:50:32. 3959 ] GET /healthz: (1.0472ms) 500 goroutine 2352 [running]: (respLogger).recordStatus(0xc001fa7110, 0x1f4) +0xd2 (respLogger).WriteHeader(0xc001fa7110, 0x1f4) +0x35 (baseTimeoutWriter).WriteHeader(0xc00382ad80, 0x1f4) +0xaf net/http.Error(0xdc0d5b8, 0xc0036a21c0, 0xc0003b5a40, 0x136, 0x1f4) +0xda (0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x3aa net/http.HandlerFunc.ServeHTTP(0xc0027a40e0, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x44 (pathHandler).ServeHTTP(0xc00278a900, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x510 (PathRecorderMux).ServeHTTP(0xc00014f340, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x8a (0x4c243b1, 0xe, 0xc0008aa360, 0xc00014f340, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x661 (0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x4c3 net/http.HandlerFunc.ServeHTTP(0xc0000da080, 0xdc0d5b8,

0xc0036a21c0, 0xc00380ef00) +0x44 (0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x459 net/http.HandlerFunc.ServeHTTP(0xc00071c9c0, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x44 (0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x1eeb net/http.HandlerFunc.ServeHTTP(0xc0000da700, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x44 (0xdc0d5b8, 0xc0036a21c0, 0xc00380ee00) +0x476 net/http.HandlerFunc.ServeHTTP(0xc0008a9400, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ee00) +0x44 (timeoutHandler).ServeHTTP.func1(0xc003637560, 0xc00045dd00, 0x6b6d240, 0xc0036a21c0, 0xc00380ee00) +0xb3 created by (timeoutHandler).ServeHTTP +0x1b0 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x25f7a93] goroutine 3306 [running]: (Scheduler).bind(0xc00002db60, 0xc004ae3c00, 0xc0049d0a80, 0x0, 0x1c4efe1) +0x643 (Scheduler).scheduleOne.func1(0xc001ffe701, 0xc00002db60, 0xc004ae3c00, 0x6b51d40, 0xc003a1c1c0, 0xc001ffe700, 0x11, 0x64, 0x64,

0xbf07eb8a4781e3e8, .) +0x21f created by (Scheduler).scheduleOne +0xb6f exit status 2 FAIL 4.409s !!! Error in ./test- Error in ./test- 'go test - -test.timeout=60m -' exited with status 1 Call stack: 1: ./test- main(...) Exiting with status 1 Anything else we need to know: run shell
/sig testing /sig scheduling
/assign


Question: const ( bridgenf                    = "/proc/sys/net/bridge/bridge-nf-call-iptables" <ins> bridgenf6                   = "/proc/sys/net/bridge/bridge-nf-call-ip6tables" </ins> externalEtcdRequestTimeout  = time.Duration(10 * time.Second) externalEtcdRequestRetries  = 3 externalEtcdRequestInterval = time.Duration(5 * time.Second)

BUG REPORT /kind bug What happened: Kubeadm checks that bridge-nf-call-iptables (IPv4) is set, but does not check for bridge-nf-call-ip6tables (IPv6) What you expected to happen: If IPv6 is being used, then Kubeadm should check for bridge-nf-call-ip6tables How to reproduce it (as minimally and precisely as possible): Just running Kubeadm, however it's only a problem when running IPv6 Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/area ipv6 /area kubeadm
/sig network


Question: linux/s390x linux/ppc64le darwin/amd64 <ins> darwin/arm64 </ins> windows/amd64 )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When trying to build kubectl from tip () on a M1 Macbook: Why is this needed: M1 MacBooks are a new major client platform that many developers will use. brew listed kubectl (kubernetes-cli) is one of the major packages still not supported on M1 () Since it's supported by go, it's only required to add it to the supported client list Locally I managed to add support using the following patch:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli architecture
/sig release
Any updates on this? cheers
you can watch the PR:
it works, nice!!!
Done!!!


Question: specs := []*spec.Swagger{} for _, v := range crd.Spec.Versions { <del> s, err := builder.BuildSwagger(crd, v.Name, builder.Options{V2: false, StripDefaults: true, StripValueValidation: true, AllowNonStructural: true}) </del> <ins> s, err := builder.BuildSwagger(crd, v.Name, builder.Options{V2: false, StripDefaults: true, StripValueValidation: true, StripNullable: true, AllowNonStructural: true}) </ins> if err != nil { return nil, err }

With in a CRD we get: This is because gnostic's OpenAPI proto types have no nullable support. Luckily CRDs keep working because in the upper error case we don't stop serving, just set the SSA schema to nil.
Thanks for reporting!
I think it should be "This is because gnostic's OpenAPI v2 proto types have no nullable support.". The problem is that we're not supposed to pass-in v3.
Source:
OK, We'll write a test and fix this, thanks Stefan!
/reopen This still happens in slightly different shape: This comes from :
That's a fairly different bug but yes, thanks!
Is this related to the stating that use of "anyOf" wouldn't be allowed after 1.16? We're seeing issue on OpenFaaS after going to 1.17.5 and 1.18. Usually upgrading K8s versions hasn't caused breaking changes and we're trying to figure out where to start with this one.
No. That post was about structural schemas. v1 CRDs are limited to structural schemas. v1beta1 CRDs can continue to create non-structural schemas until v1beta1 is no longer served (in 1.22) Also, is not completely disallowed, specific restrictions inside an clause (like ) are what are not allowed ("logical constraints using oneOf, allOf, anyOf, not are allowed").
extracted the newly found issue to to track separately with isolated reproducer /close
Closing this issue. <detailsIn response to : extracted the newly found issue to to track separately Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } fakeDocker.containerList = []docker.APIContainers{ { <del> // format is <container-id>--<manifest-id> Names: []string{"bar--foo"}, </del> <ins> // format is k8s--<container-id>--<manifest-id> Names: []string{"/k8s--bar--foo"}, </ins> ID:    "1234", }, { // network container <del> Names: []string{"k8snet--foo--"}, </del> <ins> Names: []string{"/k8s--net--foo--"}, </ins> ID:    "9876", }, }

release-master-blocking Conformance-GCE-master-kubetest2Kubernetes e2e suite.[It] [sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance] ci-kubernetes-gce-conformance-latest-kubetest2.Overall kubetest2.Test 07-11 15:51 EEST No response /sig cloud-provider cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I remember some timeout has changed with ginkgo2 and something moved from 24hours to 1 hours, or something like that, is it possible that this is the problem here, in this job?
The Grinko timeout change could also be the root problem with this job:
hmm, we have to revisit the flags
hmm, this seems correct
that was a good observations, seems that job is failing because gathering the logs takes too much time !
there is no timeout set in the kubetest2 command here (not sure if this would be the place to set a timeout)
it seems kubetest has to be migrated to ginkgov2 it should use this as basis I'm on vacation this week, can somebody take over and open the issue and patch kubetest2?
looks like this file needs to get updated: add line 72:
oh, you already commented this :D
I will have a look later
Related: umbrella issue:
last testgrid run passed :)


Question: // Flatten the list of current endpoint infos to just a list of ips as strings curEndpointIPs := flattenEndpointsInfo(curEndpoints) if len(curEndpointIPs) != len(newEndpoints) || !slicesEquiv(slice.CopyStrings(curEndpointIPs), newEndpoints) { <ins> glog.V(3).Infof("Setting endpoints for %q to %+v", svcPort, newEndpoints) // Gather stale connections to removed endpoints </ins> removedEndpoints := getRemovedEndpoints(curEndpointIPs, newEndpoints) for _, ep := range removedEndpoints { staleConnections[endpointServicePair{endpoint: ep, servicePortName: svcPort}] = true } <del> glog.V(3).Infof("Setting endpoints for %q to %+v", svcPort, newEndpoints) // Once the set operations using the list of ips are complete, build the list of endpoint infos proxier.endpointsMap[svcPort] = proxier.buildEndpointInfoList(portsToEndpoints[portname], newEndpoints) </del> } <ins> // Once the set operations using the list of ips are complete, build the list of endpoint infos newEndpointsMap[svcPort] = proxier.buildEndpointInfoList(portsToEndpoints[portname], newEndpoints) </ins> activeEndpoints[svcPort] = true } } <del> // Remove endpoints missing from the update. </del> <ins> // Check stale connections against endpoints missing from the update. </ins> for svcPort := range proxier.endpointsMap { if !activeEndpoints[svcPort] { <ins> glog.V(2).Infof("Removing endpoints for %q", svcPort) </ins> // record endpoints of unactive service to stale connections for _, ep := range proxier.endpointsMap[svcPort] { staleConnections[endpointServicePair{endpoint: ep.ip, servicePortName: svcPort}] = true } <del> glog.V(2).Infof("Removing endpoints for %q", svcPort) delete(proxier.endpointsMap, svcPort) </del> } proxier.updateHealthCheckEntries(svcPort.NamespacedName, svcPortToInfoMap[svcPort]) } <del> proxier.syncProxyRules() </del> <ins> if len(newEndpointsMap) != len(proxier.endpointsMap) || !reflect.DeepEqual(newEndpointsMap, proxier.endpointsMap) { proxier.endpointsMap = newEndpointsMap proxier.syncProxyRules() } else { glog.V(4).Infof("Skipping proxy iptables rule sync on endpoint update because nothing changed") } </ins> proxier.deleteEndpointConnections(staleConnections) }

3 node Kubernetes v1.2.4 cluster with HA scheduler and controller-manager have no additional POD running aside from those in the namespaceruns twice per second. Increasing verbosity shows that this is triggered by Stopping all the schedulers and controller-managers, then restarting only one instance of each without stops from running that often.
That seems to be related to
We should try to filter updates to the proxy when the only change is an annotation that it's not watching. This seems to be an unintended consequence of leveraging endpoints.
I'm going to use this one as the root issue.
Do the frequent restores cause any problems?
Performance wise we've had several issues... they can be computationally expensive the larger the number of routes you add.
It's fairly anecdotal, but during the brief time these restores are running, they seem to consume a fair amount of CPU. Especially on an already very heavily-loaded node, this is undesirable.
We should be able to calculate when a restore is a noop and not run the restore. is anyone currently working in this area?
To really know whether an iptables sync is needed you have to do all the work except actually updating it. Working around this in iptables seems like a bad answer - this is delivering 2 events per second per node. Is that really necessary?
It's one update every two seconds by default and it's configurable. By increasing that period, you make a tradeoff with responsiveness of cluster failover. Conceivably updates in large clusters are much more frequent.
maybe Endpoints was the wrong object to watch? , Mike Danese wrote:
From an API perspective, annotating Endpoints, a collection of pods that implements a service, with which endpoint in the collection is the current master makes sense. I don't think anywhere else makes as much sense except for possibly a seperate resource. It's unfortunate that the kube-proxy cannot handle frequent updates to fields that should be noop on these objects. If we can't fix that generally then it shouldn't be hard to special case a fix for the leader election annotation. I'd still like someone to quantify what the effects of these frequent updates are e.g.: frequency/duration/magnitude of utilization spikes whether that utilization is spent traversing the state data structure in process or spent in fork execs This info would be very helpful in priortizing this.

There are only a few objects that are watched by every single node in the cluster. I know it makes sense to be there, but it really is sort of a pessimal choice. , Mike Danese wrote:
We were in favor of component registration, but that seems to have died.
! As you can see from the output of here (rows highlighted in pink), / are using a non-trivial amount of resources over a given 10 second interval; ~1.5 seconds of CPU time and 42 processes launched. You said you wanted to know what was happening internally, but given the number of processes – there were 14 invocations of and 28 of – it's difficult to know precisely what I should . In any case, for something that is essentially a no-op, this activity seems excessive and undesirable.
So we have a BZ open against too many iptables refreshes on heavily loaded clusters but all the comments are private: . The distillation is it's eating time in the kernel (kernel-commit [net] netfilter: xtables: speed up jump target validation" that speeds up iptables operations somewhat.) However, there are still issues with incessant refreshes + large number of rules.
If someone wants to make the kube-proxy logic smarter in the face of no-op changes, I'm happy to entertain that patch, but I am more concerned thatthis is broadcasting events to every node every couple seconds. , Timothy St. Clair
imo we need to lock on a different object, at 5-10k nodes this is an untenable level of overhead.
The anecdote indicates that the immediate perf issue is caused by iptables-restore. In small clusters we can alleviate this by not restoring noops. I'd like to see the frequency of updates in large clusters in regular state (e.g. pods dieing, deployments in progress). Annotation updates may be a small portion of total endpoints updates. If endpoints updates are happening anyway and they are batched, then moving off of endpoints may not be useful. If these updates are not noops then optimizing out noop iptables-restores will also not be useful. I'd be happy with moving to another resource if we identify a better place for this. I'm also happy to revisit the default config for the leaderelection client.

I'd be unhappy with a change that diminishes the clarity of the API on an assumption that it will improve performance. It's possible that the tradeoff is worth it but we need to validate/quantify that assumption.
They are batched, but the watch updates will still occur across the cluster. So the more components we have annotating different endpoints the larger the fan-out. Just the notion of a single annotation could cause 10k watch updates for actors who don't care and then have to filter gives me pause. /cc
given that 3rd party api is now in place, would you have any objections if I were to make a pull to allow locking there?
nope, sounds fine.
Exiting crazy release+replanning, part 1 is available for review, onto part 2.
Third party resources don't have validation or conversions across versions. You should probably consider making this a first class API object to gain those benefits. Otherwise things may get really strange across version migrations.
So the only other object that seems appropriate might be component-status. thoughts?
I definitely think that ComponentStatus is better than Endpoints.
k, making change..
SGTM, but IIRC, component status is not backed by etcd?
I just found that out a second ago :-(
So were kind of in a bit of a conundrum. 3rd party doesn't work, and component status isn't stored in etcd. So I see 2 options: the logic for querying componentstatus into something like the node controller and reflect that data etcd. to revisit a scaled down version of component registration to address add-ons in the kubeadm world? Thoughts? /cc
When last I looked at it, the component stuff was heading in a direction to be a convention on top of services/endpoints. That seems right to me -- it is a way to move away from a special type of endpoint that was rushed in a while ago. That doesn't solve the problem of updating leader information. My initial read (without studying this deeply) is that spamming updates to the endpoints object is the wrong way to do this. Can we only update the endpoints object once a new leader is elected and do the spammy stuff elsewhere? (I think that is what the discussion about third party resources used for this is).

At the end of the day, is this something that kubernetes should be involved in? If we had a lock server that everyone could talk to (with a security identity and quotas) then that could be used for leader election. Do we want to turn the api server into a generic lock server too?
Only for integrated components, or add-ons, of kubernetes. If we're going to self host the control plane, it's now an implicit requirement if you want HA, otherwise you're adding another etcd which would need it's own setup+config.
What would be really nice is if we could filter a watch based on a field path and only recieve updates if those fields are updated. We need some place to store the lease record. Every resource in kubernetes is a already a lock.
I don't want this to fall off the rails. So it sounds like Option 2 is thorny, but definitely an issue to address as add-ons/components grow. We need a secure reliable mechanism to register with the system and stating leader information is a pretty natural thing to do with such a facility. In the long run, I still believe it is the correct solution, but I doubt we are going to get there soon. That being said the issue presented here is an immediate concern and I don't see any major down sides to Option 1 listed above, but I'd need to do it in a couple of parts.
xref (the component status PR ) which is lost forever because appearantly chokes on it?
I only just realized this was happening after I think the real cost of kubeproxy restoring rules multiple times a second is increased iptables lock contention (and cpu). It might stall concurrent kubelet iptables modifications for some smallish period. I'm inclined to not put in a spot fix for kubeproxy, but I want to make sure it isn't an issue, ?
Is it a spot fix to not call restore if there is no diff in the rules? How hard is it to calculate this diff? That doesn't resolve this issue but it seems like it would mitigate one of the bad symptoms.
I think it should be easy. Definitely should make 1.5.

If no ones impacted by this "hotlooping", cherrypick into 1.4.X will depend on exactly how easy the fix is, if people are impacted, we should fix and cherrypick regardless. I'll need to poke around a bit.
I don't agree with the fix suggested by Basically the problem is that every "Endpoints" object is watched by all KubeProxy in the cluster. Which means, that if we are renewing the lease every 2s (which IIUC is the current case), and we have 2000-node clusters, that means that 2000 KubeProxy components are processing that Endpoint update every 2s. No matter if we will be updating iptables in that case or not, this is clearly a waste of time. For leader election we should use an object that components that run on nodes don't listen to - this in my opinion is pretty important requirement in this dicussion.
So I need "buy-in" on the proposed fix around pushing component status into a etcd-backed resource which it currently isn't. Are there any oppositions to this?
If you want to make it so kubeproxy doesn't get a watch event that it needs to ignore, less work for me :) I was proposing something in the here and now, given that this is happening in live 1.4 clusters. I'm guessing component status is not landing in time for 1.4.2.
yeah - I'm not sure if we even cherrypick it at all. So if we want to fix it in 1.4 branch (do you think we want - I don't know how important it is), then probably what you described should be a temporary solution. - regarding buy-in, I don't have any strong objections about it. Though may have some. WDYT?
I haven't seen this until now. What component is using the endpoint objects to do locking?
I would prefer to see us provide pods a way to present any information currently in componentstatus in their own pod status, and get rid of componentstatus entirely. It's a service we should provide for all components, not just system components.
Short term, assuming the locking behavior is legit, either kube-proxy needs to filter out fields it doesn't care about, or we need to get apiserver to filter out such changes (better but much more complex).

If kube-proxy is relying on the system to minimize the number of update messages it gets, that's a mistaken assumption in kube-proxy.
here's another user for your alternate view proposal.
Controller manager and scheduler have done this since 1.2.
That should affect a vanishingly small percentage of endpoints, so not much point to doing the extra work to fix it in apiserver. IMO this is just a plain old kube-proxy bug. , Mike Danese wrote:
kube-proxy can be smarter for sure, but this still broadcasts a useless change to every node every 2 seconds. It really should be a different object that is less likely to be actually used. IMO , Daniel Smith wrote:
+1000 - this is exactly what I was trying to express in
+1000 to and , The fan-out at large scale is simply untenable. So on the scale of a 10k node cluster where we had say a minimum of 2 components being updated every 2 seconds that is 10k x (60/2)*2 = 600,000 broadcasts in a minute even with filtering on the proxy side. I for one am a fan of open transparent communication, but this may be a bit too much ;-).
Ah, now I see what you're talking about. I agree it sounds bad, but is it a big percentage of the endpoints traffic at that scale? This costs 10k broadcasts per second. Assume there's 50 pods per node. Assume each pod shows in one service. Assume a pod changes status once every thousand seconds (most pods won't change often, but pods get started/stopped and crashloop often enough to raise the average). 50  10k  .001 = 500 changes per second, makes 5,000,000 broadcasts per second. So the number you are focusing on is like 1% of the number of endpoint changes at that scale... Maybe my numbers are off above, but are they off by two orders of magnitude? I doubt it. So really, you are running into a scaling problem that's fundamental to endpoints, and none of the above solutions will really address it.
I think there's actually a very old issue about endpoints not scaling, probably knows off the top of his head.

(Note that I don't object to changing scheduler and controller manager to use a different object for locking--endpoints was a weird choice IMO--but it is only adding a few feet onto the end of your runway.)
DO NOT persist componentstatuses in etcd. That API is a disaster. See for discussion of its future directions. I don't think endpoints scalability has its own issue, though it has been mentioned in a bunch of places: Pagination is
Ah, I may be misremembering IRL conversations as github issues... , Brian Grant wrote:
Endpoints as an API object has scalability problems (needs some form of pagination) and kube-proxy should batch updates (and filter no-ops). I am 99% sure both of these have open issues. , Daniel Smith wrote:
I don't see how pagination addresses the fan-out problem, or any problem. Even in a 10k node cluster there's only 500k - 1m pods total, and those will be merged into fewer endpoints objects, maybe 2x to 1000x fewer, depends on how big the services are. I think adding a low-cost endpoints view using the mechanism Clayton proposed is not enough; we can filter out metadata changes, ordering changes, and inactive endpoint changes, which kube-proxy doesn't care about (right?). I think that, at best, that filtering will only be 2-3x reduction, which is not enough. So intelligent update batching will be required, or we need a different distribution mechanism. (kube-proxy binaries could do some sort of p2p endpoint distribution, for example.) , Tim Hockin wrote:
(The right kind of update batching would require extensive changes to our watching model, in case it wasn't clear.)
Pagination is about the 1M etcd limit - at ~16 bytes to store an IPv4 address in JSON we will overflow that object after 32k-64k endpoints in the best case, potentially much less in the worst case. I know that feels like a lot, but I'd rather not get surprised by this one. This is not unique to endpoints. , Daniel Smith wrote:
If you have lots of services then, yes. That said I don't think it's super urgent problem. - do you have the same limit for size of value in etcd v3?

Guys, I'm sorry to jump in here with little context, but we need some solution to this issue in the short term. This is seriously impacting the performance of our nodes in production, and from my perspective it would make sense to add a fix that addresses the immediate issue now (probably in kube-proxy), and then separately make the endpoints API better in the medium term. It would be a shame for this issue to sit for ages while we debate the proper solution. Does this make sense? I'm very happy to pitch in and write some code if necessary
Above volunteered to look into the kube-proxy performance issue. Thanks, Beeps.
How about a configmap?
I was waiting for someone to complain (). Can you clarify what exactly is the issue? () just cpu or are you running something else that needs the iptables lock? kube-proxy should already have a request: 100m, meaning that's what it gets if there's contention, and if there isn't it can take how much it needs.
forked, this issue can be about node local implications of spammy watch notifications for kube-proxy
That's not pagination, that's sharding the endpoint object, which may not be necessary in etcd3 on account of size. I'd sooner worry about write contention with that many endpoints. (Pagination is where apiserver serves a list request in multiple pages.)
I guess my point is more that if kube-proxy / iptables can't handle the 2 second updates without being a CPU hog, it's going to have major problems in even medium sized clusters. Is there something odd about OP's setup? Why aren't other people complaining? Why does kube-proxy update iptables as fast as it can, why not batch and update every 5 seconds or 15? Can it precompute something and then take a lock for less time?
Yeah I think there's kubeproxy optimization and unnecessary watch notifications from overloading an object broadcast cluster wide. That's why i forked the bug.
They're not paying close attention to node performance profiles...? Also, we do HA differently so it never jumped out in our profiles, except during bulk operations. (see comment below) Broadcasting is still an issue, but we've proposed a similar approach:
Yes, the issue is CPU performance. In my you can see the sort of CPU is consuming.

On heavily-loaded nodes, this penalty and the associated context switches are not insignificant. I'm not sure why others are not seeing this issue to be honest; we are running HA master, on AWS, with some quite heavily-loaded worker nodes. I don't know how common this is, or as points out, how much attention others are paying to their node performance. It's also possible that we are doing something peculiar, but I can't think what. If you want me to gather more data about the performance impact this has I can, but I think I can say with certainty that this is a real problem.
Hmm, in that comment it looks like you have nothing else on the node. If you're saying it's getting much more than its "request: 100m" even on a loaded node, that sounds like a different bug. Of course we should still fix it, I'm just wary of pointing fingers at something that has a relatively low request.
Agree, there's an open issue for proxy to batch updates. There's really no need for it ever to run more than once per second, and even that is probably too much. , Daniel Smith wrote:
FTR - I did try to to repro this and had kube-proxy writing out thousands of iptables rules as fast as it could, and I never saw a single iptables-restore more than 120ms or so, (numbers from memory) , Prashanth B wrote:
/cc - fyi...
Since nobody mentioned this yet: the sheer amount of logging is causing an issue in itself. Is it really necessary to write out iptables this verbose? Also in journalctl it all shows up in red suggesting it is sent to stderr which made me feel uncomfortable looking at this for the first time. I searched for errors, but there were none.
IIRC you said you had a 3.x something kernel, while all our customers are either using later kernels, or using a highly rebased network stack from later kernels. I was able to reproduce the issues with a 4.7 kernels on Fedora. I haven't tried recent Ubuntu. But in any case, iptables-restore latency is highly kernel dependent it seems.
What verbosity are you running? The normal setting is --v=2 and that does not log all the iptables state.

That is --v=3 , leo-baltus wrote:
It sounds more and more like a kernel regression , Tim Hockin wrote:
Ah, running with --v=4. I wasn't aware of this, this is a setup made by kubeadm. Sorry for the noise. Op 25/10/2016 om 11:19:28 -0700, schreef Tim Hockin:
So if the fan-out is the problem, perhaps we want publish/subscribe instead? A message queue?..
cross-linking . We are a bit behind on versions (still on 1.3) and at least there, there are real updates to the default/kubernetes endpoints object. These come from the apiservers each thinking they're the one and resetting the endpoints list. suggests that there is a flag that needs to be set to remedy that but there is no documentation?
I'm trying to troubleshoot networking issues in my cluster by watching iptables packet counts using . It looks like a symptom of this issue is that the packet counters are continuously being reset by reloading iptables. Also, the rule order also gets updated continuously which makes it difficult to focus on one rule. Am I correct in assuming this issue is the cause?
probably, yes. , ravishivt <:


Question: <del> #!/bin/bash # Copyright 2014 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Bring up a Kubernetes cluster. # # If the full release name (gs://<bucket>/<release>) is passed in then we take # that directly.  If not then we assume we are doing development stuff and take # the defaults in the release config. # exit on any error set -e source $(dirname $0)/../kube-util.sh echo "Starting cluster using provider: $KUBERNETES_PROVIDER" verify-prereqs kube-up # skipping validation for now until since machines show up as private IPs # source $(dirname $0)/validate-cluster.sh echo "Done" </del>

Rackspace support has not ben updated in a few months. cluster/rackspace/cloud-config/minion-cloud- is using etcd_servers for kubelet and kube-proxy, but support for them has been removed since v0.9, I think. I'll remove it in a few days if no-one makes a counter-offer to get the config working with latest release.
There's been some work on this but it is blocked while we figure out some load-balancer design stuff. , Eric Tune wrote:
Okay, I will remove the link to rackspace until it is fixed (). Leaving this issue open to remind us to delete it in, say, a month, if no progress is made.
cc
See also
I'd prefer to try to get things updated and keep support. However I don't want to continually become stale. every few months either.
Hi all, I want to use kubernetes on my existing rackspace infrastructure but I don't understand if rackspace suppor was removed or not. In Doc, under "custom cloud solution" it's linked but I wasn't able to deploy it neither with install method nor with . With i got a error (I have 16GB or RAM!). thanks in advance, Fx
It looks like no support has happened on Rackspace for a year.
You should not expect it to work, and we should delete rackspace support.
ACK.
Unfortunately I think we'll need to drop support for Rackspace as a deployment target since I can't commit to keeping the deployment scripts up to date. Rackspace support as a cloudprovider should be maintained.
I would be willing to support the Rackspace deployment target, if there is interest to keep it alive.
There would be definitely an interest from our side - sad to see that Rackspace has no interest in maintaining it
No problem, I will take over the maintenance. It will take me a bit of time to get caught up, but I will get it done as soon as possible.
Do you think it would be hard to get the of LBaaS working ?
Sorry it took me a bit to respond. there is a quick update as to where I am. I was busy with some stuff, that prevented me from working on this. But that has been resolved, and should have updates by the end of next week.

However what's causing the delays is the lack of a good API for Rackspace Cloud. I've used the gophercloud, it works but I'm not a fan. Additionally gophercloud doesn't have full API support (one of the things I'm not a fan). This would make it hard to implement LBaaS. I should have a better idea of your request sometime next week.
I have seen this api docu: - is it the same as you saw? I think it should be doable with it.


Question: "", // Default fsType ), Capabilities: map[testsuites.Capability]bool{ <del> testsuites.CapExec: true, </del> <ins> testsuites.CapExec:             true, testsuites.CapSingleNodeVolume: true, </ins> }, }, }

What happened: When we run csi hostpath driver e2e tests on a multi-node cluster some tests failed because they assume that volumes are node independent but this is not the case for some drivers like hostpath. Related PR; What you expected to happen: Skip the tests that assume volumes are node independent for the drivers that don't support node independent volumes. /sig storage /assign


Question: err = framework.WaitForJobFailure(f.ClientSet, f.Namespace.Name, job.Name, framework.JobTimeout, "BackoffLimitExceeded") Expect(err).NotTo(HaveOccurred()) <del> By("Checking that only one pod created and status is failed") </del> <ins> By(fmt.Sprintf("Checking that %d pod created and status is failed", backoff+1)) </ins> pods, err := framework.GetJobPods(f.ClientSet, f.Namespace.Name, job.Name) Expect(err).NotTo(HaveOccurred()) <del> Expect(pods.Items).To(HaveLen(1)) pod := pods.Items[0] Expect(pod.Status.Phase).To(Equal(v1.PodFailed)) </del> <ins> // Expect(pods.Items).To(HaveLen(backoff + 1)) // due to NumRequeus not being stable enough, especially with failed status // updates we need to allow more than backoff+1 // TODO revert this back to above when https://github.com/kubernetes/kubernetes/issues/64787 gets fixed if len(pods.Items) < backoff+1 { framework.Failf("Not enough pod created expected at least %d, got %#v", backoff+1, pods.Items) } for _, pod := range pods.Items { Expect(pod.Status.Phase).To(Equal(v1.PodFailed)) } </ins> }) })

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: for a Job does not work on Kubernetes 1.10.0 What you expected to happen: can limit the number of time a pod is restarted when running inside a job How to reproduce it (as minimally and precisely as possible): Use this resource file: If the job was created in Kubernetes 1.9, it will soon fail: While creating the job in Kubernetes 1.10, it will be restarted infinitely: Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.0 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Kubeadm Others:
/sig scheduling
/sig job /sig lifecycle /sig apps
/sig testing
We just updated to 1.10 and seem to be affected by this, too.
I had to add activeDeadlineSeconds to all Jobs and Hooks, as workaround...
Can confirm with 1.10.1
It seems to be introduced by cc
Cause appears to be that the method gets exactly one attempt to update the status with the failed condition and if it fails due to resource conflict, it never adds the condition. The controller code seems to use requeues to track failures, but if a sync loop executes successfully, it would lose track of the failed item ever having been requeued. Looking into a fix now.
Looking at , we force an immediate resync when we see a pod that has failed which also clears the key in our queue, but that in turn makes us lose state of the number of requeues.
This bug killed my test cluster over the long weekend (with + pods). Luckily we don't yet use CronJobs in prod.
killed my cluster too :( still existing in kubernetes 1.10.2
This issue also can't be mitigated as terminated are not part of the resource quota:
This is a pretty terrible bug and it definitely still exists in v1.10.2
/priority important-soon
can you please take a look?
Looking right now...
I've opened to address the issue. Sorry for the troubles y'all.
pulled the patch into current 1.10.2 release and it works! Thanks :)
I filed (with and a test for Job with backoffLimit=1).

The test exposes some other problems. Here are two failure mode: never retries (): the code uses to see the number of retries. However, the job can fail to be synced just because of an update conflict (). That shouldn't be seen as a retry. retries more than needed (): Job controller creates more pods than needed because it doesn't realize itself has failed and exceeded backoffLimit earlier. This may be due to stale cache or that the job status failed to be updated earlier. I think Job backoffLimit is still broken.
FWIW,(assuming I'm not missing something) I can confirm on GKE 1.10.2-gke.3 (latest as of Jun 2nd) the is still broken. The following job spec endlessly creates pods:
we used the latest v1.10.3 release but backoffLimit is still broken
This issue prevents us from rolling 1.10 to all clusters
/milestone v1.11 Looks like we have a couple of PR(s) in progress and feels like we should get this fixed by the time we ship 1.11, please let me know if this fine or not.
I think that this issue should be fixed in 1.10 regardless of 1.11 as this issue causes serious cluster stability, for a fairly common scenario. Specifically for us, migration to 1.11 adds complexity and risk due to the change in the way nvidia GPUs are managed, so it will take us some time to upgrade to 1.11.
y, guess my question is ... is it fixed in master? (and hence already fixed for 1.11), if not, do we need to get this fixed in master/1.11 and backport to 1.10
Unless I'm mistaken, the objects are considered stable APIs right? (since they are under the group). If so, I also think this should be fixed in the next 1.10 point release. In general, like we would also rather to be able to use s without being forced to upgrade to 1.11.
This is a regression in functionality, and when the proper fix is determined, it should be cherry picked back to 1.10. We need it fixed in master first, however. I'm going to escalate this to sig-apps to see if we can get the right resources.

cc: This will need to merge into master, and then be cherry picked back to 1.10
PTAL
The fix is ready for review and will be cherry-picked for the next 1.10 release.
Reopening for cherry pick.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I didn't see anything in the Changelog for v1.10.4, so i'm guessing this still hasn't been fixed yet?
The fix has landed to master but not to 1.10 yet. See It will get hopefully merged before the next 1.10.x release.
Having the same issue. Waiting for update.
This fix has been cherry picked back to 1.10, and should show up in the next patch release (1.10.5).
Any ETA for 1.10.5 yet? Even a ballpark date would be fine.
I'm planning to cut 1.10.5 tomorrow, assuming all tests will be green. In case of any last minute problems it may slip by a day or two, but so far everything looks good for tomorrow.
Thankyou to all involved! I really appreciate this being rolled back into 1.10.
Given that the fix is already on 1.11 and on 1.10 branches, this issue should be closed. Thanks for fixing this!
/close
I don't believe this is fixed? Unless I'm doing something wrong.. Both resource specs have a and . This is in GKE,
I have the same bug on 1.9, with an openshift cluster: Any plan on backporting it to 1.9 as well?
you likely are encountering a different issue, as the root cause of this bug was introduced in kubernetes 1.10, not 1.9.

Also, kubernetes 1.9 is now outside of the support window, so no fixes will be backported to the upstream branch.
I also find this bug in kubernetes 1.12.1 Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} In pod failure condition,when i set parallelism 2, backofflimit 0,it will destory the last failure pod,only left one pod remain;And set parallelism 2, backofflimit 1,sometimes it keeps 3 pods,sometimes it kills the last failure pod remains 2
@ kubectl version: Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} minikube version: v0.30.0 apiVersion: batch/v1 kind: Job metadata: name: whalesay1 spec: template: spec: containers: name: whalesay image: docker/whalesay command: ["cowsa", "This is a Kubernetes Job!"] restartPolicy: Never backoffLimit: 2 NAME READY STATUS RESTARTS AGE pod/whalesay1-6lt4b 0/1 ContainerCannotRun 0 59s

pod/whalesay1-86tlz 0/1 ContainerCannotRun 0 48s pod/whalesay1-8v8nr 0/1 ContainerCannotRun 0 1m pod/whalesay1-8w9k2 0/1 ContainerCannotRun 0 1m pod/whalesay1-8xxtr 0/1 ContainerCannotRun 0 53s pod/whalesay1-dl4l8 0/1 ContainerCannotRun 0 1m pod/whalesay1-fhlhg 0/1 ContainerCannotRun 0 37s pod/whalesay1-fz8mv 0/1 ContainerCannotRun 0 26s pod/whalesay1-g2mt6 0/1 ContainerCannotRun 0 9s pod/whalesay1-hgj95 0/1 ContainerCannotRun 0 1m pod/whalesay1-hrsrd 0/1 ContainerCannotRun 0 1m pod/whalesay1-hzm42 0/1 ContainerCannotRun 0 1m pod/whalesay1-mvctw 0/1 ContainerCannotRun 0 1m pod/whalesay1-n7zq9 0/1 ContainerCannotRun 0 2m pod/whalesay1-nl89l 0/1 ContainerCannotRun 0 31s pod/whalesay1-nqcmg 0/1 ContainerCreating 0 2s pod/whalesay1-p85rm 0/1 ContainerCannotRun 0 2m pod/whalesay1-pjzmd 0/1 ContainerCannotRun 0 2m pod/whalesay1-q2k7p 0/1 ContainerCannotRun 0 43s pod/whalesay1-rhvz7 0/1 ContainerCannotRun 0 1m pod/whalesay1-rzrtg 0/1 ContainerCannotRun 0 1m pod/whalesay1-xp5tr 0/1 ContainerCannotRun 0 1m pod/whalesay1-zn2s6 0/1 ContainerCannotRun 0 1m pod/whalesay1-zq4g9 0/1

ContainerCannotRun 0 14s pod/whalesay1-ztxdv 0/1 ContainerCannotRun 0 19s NAME DESIRED SUCCESSFUL AGE 1 0 2m
Just update to something newer than 1.10.0. That version is ages old and I'm sure it is fixed within the latest 1.10.x release. (Which is also out of support so update to 1.12/1.13 ASAP.)
In version "v1.10.11" the though set to 4 for a Job's pod, the pod eviction happens for 5 times and at few times it was for 3 times. Also to note that we also have applied and Below is the version details of Kubernetes that we are using
Though we have set the backoffLimit of a Job to 4 the pod eviction is still being done more than 4 times i.e. 5 times at one of the instances its three times as well. The eviction of the pod happens for a geniune reason i.e. the disk space of the pod node exhausts when the application within the pod executes, we know the reason why that's happening. The point is that the Job's-Pod should be attempted 4 times and not 5 times. Please note that strangely we don't get to see this error in another environment where we have v1.10.05. providing the version details of that environment below There the pod eviction happens as per the stated literature of Eviction with BackOffLimit. We have no issues there. Now when we have upgraded to v1.10.11 recently [ About two days back ] we are getting the reported problem. Would appreciate if someone in this forum can advise on the root cause and a solution. Do we have any flaw with version "v1.10.5" like security loophole.
10.11 is required to avoid a critical security bug. I imagine we will need to update to fix this, but we will need to find out which of the patched versions has the fix we need. We can then roll this through the environments as a staged update.
As noted above 1.11+ are supported by the community until 1.14 is out:
I experienced the same behavior with . Can somebody confirm that this not happening with = versions?

We are experiencing the same issue with version 1.11.9-docker-1. If it makes any sense we have 2 enviroments. One is Ubuntu 18.04.2 LTS and the other is CentOS Linux release 7.5.1804. It happens only at the Ubuntu enviroment.
In meantime we updated cluster: We still experiencing the issue.
Hi everyone! This issue is for a specific problem in Kubernetes 1.10. Right now, the upstream Kubernetes project supports the following three versions: 1.13, 1.14, 1.15. If you are experiencing a similar issue under any of those three versions, then please open a new issue and provide all the details requested. As the specific bug in this issue is resolved and on an unsupported version, I'm going to lock this closed issue. Thanks!


Question: func verifyPackUnpack(t *testing.T, podNamespace, manifestUUID, podName, containerName string) { container := &api.Container{Name: containerName} hasher := adler32.New() <del> data := fmt.Sprintf("%#v", *container) hasher.Write([]byte(data)) </del> <ins> util.DeepHashObject(hasher, *container) </ins> computedHash := uint64(hasher.Sum32()) podFullName := fmt.Sprintf("%s.%s", podName, podNamespace) name := BuildDockerName(manifestUUID, podFullName, container)

Is this to follow pointers?
Context: The follows pointers when printing. Unlike JSON marshaling/unmarshaling it also handles private fields.

I am experiencing some weird issues when enabling HTTP liveness probes for some containers. I configure the following for all HTTP-serving containers (this is part of a template used by a replication controller with a single replica): When this is configured, the following issues start to occur: One pod is already running without problems. When I add another replication controller which creates a new pod (also with the liveness probe set), the first pod (the one that was already running) is restarted. This happens even when the two pods are in different namespaces. But when a pod's HTTP server fails completely (connection to the port is refused), the liveness probe will not trigger and the pod will not be restarted. This behavior occurs consistently when I start with no replication controllers and then add them one by one. When I disable the liveness probes on the affected pods/templates, they are no longer restarted in this manner.
You say you're starting two replication controllers-- can you verify that their label selectors do not intersect?
Yes, each replication controller has a selector over a single label in the form of so the labels do not overlap. Also, this problem occurs even when the apps are in different namespaces and goes away when the liveness probe is removed.
Thanks-- last question-- Do the pods have to be scheduled on the same node for this to occur? Your second point I believe is a problem here: we treat an error during HTTP GET as healthy for some inexplicable reason. I can't really explain your first observation, unless perhaps the pods and containers have the same names, but in different namespaces AND kubelet has a namespace handling bug?
cc
"Unknown" != "Healthy" I think it's an open question about whether we want to treat a GET failure as "Unhealthy" we certainly could... --brendan , Daniel Smith wrote:
Giving this to Saad. I think it's pretty clear that if kubelet can't do an HTTP GET, then the pod isn't healthy-- they're on the same host, HTTP get can't fail for network reasons. Our internal system makes this assumption. (I broke it once!)
tl;dr... When a second replication controller is the hash of the container of the first replication controller changes causing it to be killed and recreated. The hash changes because the LivenessProbe pointer changes. Need to figure out why that happens.

Spent time trying to understand why creating two replication controllers, both with http liveness probes, causes the container from the first replication controller to be killed and then recreated when the second replication controller is created. It turns out, as soon as the second replication controller is created, the hash of the container from the first replication controller changes and thus the container is killed: Let's see how the expected and actual hashes are computed and how they end up differing. The actual hash is pulled from the docker container name (dockerContainers.FindPodContainer): The expected hash is created from the object. Specifically, this code (): Expected hash before second replication controller is : Expected hash after second replication controller is : The difference between the container before and after is: So the hash changes because the LivenessProbe pointer in the object changes. Need to figure out why this happens.
Great detective work-- I think it's clear that we shouldn't be including pointers in our hash, so we need to revisit that and explicitly hash only the things that we care about.
Adding second replication controller changes the pointer of LivenessProbe in container config is another mystery. We need to make sure not randomly update other fields too.
Dug in to this some more: confirmed that the value of the original container's LivenessProbe doesn't actually change when the second replication controller is created--only the pointer to the LivenessProbe object changes. It turns out that the instance of the  object being used to compute the expected hash is not guaranteed to be the same instance for the lifespan of the actual docker container. In fact, every time there is an etcd update (e.g. a new pod is created) a new boundpods object is created and although the values of the existing pod/container container objects don't change, because new instances are created the pointers do change (by design). So the fix for this is to ensure that the expected hash is computed only on container values not pointers. One way to ensure this would be to make sure the container object specifies no pointer fields. However, LivenessProbe and Lifecycle, since they are optional, must be able to be set to . Thus, pointer fields can't simply be converted to value fields. Another approach would be to only hash some subset of the values (maybe the name and image) of the container.

However, containers should be destroyed/recreated (meaning the hash must change) if the actual values of any of the container fields changes (e.g. the cpu, memory, volume mounts, liveness probe path, etc.), not just the container name/image. Thus, pointer fields can't simply be disregarded for the hash computation. Instead the fix must modify the expected hash to be computed on the complete deep values of the entire container object (including the actual LivenessProbe values, instead of the pointer). The easiest way to do this is to marshal the container object to json and compute the hash from the resulting byte array.


Question: srcs = [ ":package-srcs", "//cmd/kubeadm/app:all-srcs", <del> "//cmd/kubeadm/test:all-srcs", </del> <ins> "//cmd/kubeadm/test/cmd:all-srcs", </ins> ], tags = ["automanaged"], )

go test ./cmd/kubeadm/... fails with: it appears to be following a "RunCmd" pattern used by e2e tests, but unit tests should not require built binaries
can work on this
has a long discussion as to why this is. These tests only get run when gets run, which will produce the needed binary and pass it in as arg. For more information, authored this and should be able to provide more details.
Yup, the target explicitly ignores these tests: and like said, runs this script as part of its target:
then the test should have build tags to exclude it from a regular run of the package
or up for fixing that?
I thought build tags only works on the OS level or if you it would ignore it completely? Wouldnt break the command as well?
I'd also tolerate skipping the test if the built binary doesn't exist
I think that is a great solution. I think in I had initially suggested something like that. If everyone is ok with that, I will go ahead and make that change.
Sounds good, make that change and we'll see how it turns out. I'd really want these tests start working, so +1


Question: fi } <del> # Verify and find the various tar files that we are going to use on the server. # # Vars set: #   SERVER_BINARY_TAR #   SALT_TAR function find-release-tars { SERVER_BINARY_TAR="${KUBE_ROOT}/server/kubernetes-server-linux-amd64.tar.gz" if [[ ! -f "$SERVER_BINARY_TAR" ]]; then SERVER_BINARY_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-server-linux-amd64.tar.gz" fi if [[ ! -f "$SERVER_BINARY_TAR" ]]; then echo "!!! Cannot find kubernetes-server-linux-amd64.tar.gz" exit 1 fi SALT_TAR="${KUBE_ROOT}/server/kubernetes-salt.tar.gz" if [[ ! -f "$SALT_TAR" ]]; then SALT_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-salt.tar.gz" fi if [[ ! -f "$SALT_TAR" ]]; then echo "!!! Cannot find kubernetes-salt.tar.gz" exit 1 fi } </del> # Take the local tar files and upload them to the master. # # Assumed vars:

I suspect we're going through an if/else clause where KUBETARURL isn't defined, and not defaulting it, though the function header seems to indicate it should be set in the callee.
Will take a look.
Is this in Jenkins, or on a local run? The Jenkins push job is borked as well, but it looks like it's for a different reason: Which I'm taking care of in .
Jobs are generating incredibly large log files (300 Megs?) with lots of repeated lines like this: Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 I think this needs to be resolved, since it already caused an out-of-disk in Jenkins.
Okiedokie. Yes, this was my fault. silently expects and to be set in . I the extra expectation that be set, but didn't think about how would then also have to set that. Working on a fix.
(For anyone who cares, culprit is .)


Question: } cap := persistentSpec.Spec.Capacity[v1.ResourceStorage] size := cap.Value() <del> if size != 100*1024*1024*1024 { </del> <ins> if size != 100*volume.GB { </ins> t.Errorf("Provision() returned unexpected volume size: %v", size) }

GCE PDs are allocated in chunks of GBs (not GiB) but function incorrectly creates volume in chunks of GiB - cc /sig storage
/kind bug
/assign
/priority important-soon
Hey I submit a pr for this issue. Your suggestions are very important. Thanks.


Question: Expect(err).NotTo(HaveOccurred()) } } <ins> func skipTestUntilBugfix(issueID string, driverName string, prefixes []string) { var needSkip bool for _, prefix := range prefixes { if strings.HasPrefix(driverName, prefix) { needSkip = true } } if needSkip { framework.Skipf("Due to issue #%s, this test with %s doesn't pass, skipping until it fixes", issueID, driverName) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Probably caused by <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
https://k8s- /assign
I understand that moved crd creation logic to e2e test's k8s cluster deployment code. On the other hand, I tested it with manually deployed k8s cluster. So, in my case, I guess that in addition to enable and feature gates enabled, I need to create crd manually before tests, like below: I could make two of three SkipAttach tests pass in my environment by above way. For "non-attachable volume does not need VolumeAttachment" case, I'm still looking into it. (It doesn't seem to skip creating volumeAttachment, properly)
I found a typo in 's argument. While hostpath driver's connects driver name and unique name without "-", it connects them with "-". This makes "non-attachable" test fail, due to the mismatch between volume name in crd and provisioner name, as a result, creating volumeAttachment isn't skipped. I will create a PR to fix it. For above bug, is unrelated. It just changed the way in how to test in manually deployed k8s cluster as I mentioned in above comment. So, there should be no problem for the PR, at least for the CI testing if it works well in CI environments.
I can verify that the CRDs are not getting installed on the alpha cluster. Investigating why.
/reopen since I see the Alpha tests failing
/milestone v1.13 /priority important-soon /kind failing-tests
/kind failing-test
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is fully resolved. There are still failures on https://k8s- /reopen
Reopened this issue. <detailsIn response to : I don't think this is fully resolved. There are still failures on https://k8s- Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you help take a look at these failures? It's testing CSI filesystem volume mode. I think it is unrelated to the crd installation issue that Saad fixed.
I tested with AllAlpha flag and manually created csi crds and confirmed that filesystem volume mode test passed in my environment and also other tests passed there. If I delete csi crds, other tests also fail, so I agree that the issue that Saad fixed is not related. However, above also means that this issue can't be reproduced in my environment.
I1109 14:56:01.141] Nov 9 14:56:01.131: INFO: At 2018-11-09 14:50:54 +0000 UTC - event for security-context--e42e-11e8-9b95-: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc--e42e-11e8-880b-" : node "bootstrap-e2e-minion-group-3wtg" has no NodeID annotation
Hostpath also seems to be failing because of driver registration issues? Not sure why other alpha tests are passing if that's the case
This test is the only test that uses  to create pod with CSI driver volume and expects the result to be success. One of the pod parameters specified there might affect this failure.
Those 2 failing tests are newly introduced in and started to fail from the beginning. Is that PR blocking 1.13? If not, can we revert and fix later? I have recently set up optional presubmit tests for alpha features at , which will pick up those 2 tests. Currently this failure is blocking
apart from the 2 failing tests, I see which addressed with is also flaky. These failures are now blocking PRs as mentioned above. As we enter code slush today, its very critical to have the PRs get in sooner than later. Can we plz revert the feature and the tests until they are fully resolved and ready to go. Thanks
I prefer disabling the specific tests instead of reverting the entire pr. We will lose important signal for csi
That works for me.

Can you cc me on the PR that disables those tests? When you re-enable the tests, remember to try them out on . :)
Let me consider if we can just skip these test for CSI, instead of reverting the entire patch.
you mean just skipping them in Alpha cluster right?
I was thinking about skipping the below test with the combination of below drivers: [Test] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode[Feature:BlockVolume] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources [Driver] csi-hostpath Won't it enough? Or, do we need to skip it only with alpha cluster?
If we just disable the tests alone and not revert the feature, we are just masking the issue. Ideally if we are not able to fix the tests and reenable them the feature should be reverted from 1.13 as well. Are you also actively working on fixing the tests and this skip is just to unblock PRs in the meantime? if so can we timebox by when we can have the tests fixed and re-enabled / the feature reverted? My fear is we might forget the fact that its disabled since there's no active reporting on it.
Actually, there is no feature in either of culprit commits. It tests or intended to improve the way it is tested. Therefore, it must be a test issue or another issue which is found by the test , not an issue introduced by the commits themselves. In the code, there is a clear comment that the test is skipped due to this issue with the reference to this issue#, so this won't be forgot. If you still worry about this issue is not tracked, we might be able to keep this issue open.
IIUC the test that were failing (and are being reverted) were for the new CSI festures going to Alpha in 1.13, right? so even these 2 culprit commits do not contain feature work, it is testing some new code additions to 1.13 and now those will be untested until we re-enable these tests. Is there an ETA to re-enable them, preferably sometime early next week? Yes +1 to keeping this issue open to track the skipped tests.
The two failing tests themselves are not for new alpha features that are in v1.13.

Instead, they are just to test csi driver works well as it works in existing in-tree driver. (And failed because they might be affected by some other alpha features or some of specific configurations in gce.) In addition, ~50 tests for csi driver, so it would be better not to remove all of the tests by reverting the PR, just for the 2 failing tests. I think that just reverting the patch won't help improving quality of k8s. As for ETA to solve the issue, I found difficulty in debuging in my local environment. It seems that the two tests fail in gce environment w/ AllAlpha=true, however, as I've already mentioned, at least csi-hostpath tests in my local environment w/ AllAlpha=true doesn't fail.(And I don't have test environment for gce.) Of course, I will help solving the issue next week. However, I will need someone's help for testing combinations of alpha features in gce environment to narrow down which flag really affects this issue. At least I would like to know the results: w/ MountPropagation=true,CSIPersistentVolume=true,BlockVolume=true,CSIBlockVolume=true all alpha flags except CSIDriverRegistry and CSINodeInfo
/reopen Thanks for the detailed explanation. I think the revert is fine in this case then. I am keeping this issue open however to track and fix the failures in these 2 tests can any of you help answer w.r.t alpha features on gce environment. thanks
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can help debug when I'm back on Tuesday. In the meantime, I would suggest looking through the failing test logs to try to triage what's going on
Here's one failing run:
I looked into the csi-hostpath failure log. As seen in the for , where the pod which has a csi-hostpath volume is being attached is scheduled, mount volume fails with these errors (1). Related code to this log is (2), so the driver doesn't seem to be registered to . As for the registration code for csiDriver, it is called as (3).

Therefore, will be logged if the driver is triggered for registration. As seen (4), registration is only triggered in , which is different from the one the pod is scheduled. In addition, driver-registrar seems to be deployed only on the node where registration is triggered (5). Is it right behaviour, while registrar is deployed as daemonset? I guess that just this fact seems to cause an issue to make csi-hostpath driver work properly. (Actually, I only tested with 1 node environment, so this might only happen in multiple nodes environment.) <details- (1) (2) Result of grep to check whether registrar is deployed to each node [ for bootstrap-e2e-minion-group-50r7][ for bootstrap-e2e-minion-group-k781][ for bootstrap-e2e-minion-group-z5cg] </details
The hostpath plugin should be deployed as a DaemonSet, so it's strange that it's only getting registered on only one node.
/assign looking into this as well
could you link the PR for skipping [external-gcepd] tests into this issue when you do that as well
skipping gcePD-external is here:
Since gce-cos-alphafeatures is now passing, do we need to keep this issue open? Should it morph into a non-blocking issue for longterm work on a better test?
We have root caused 2 of the 3 failures, which were problems in the test and test conifg. Fixes will be out soon. Still triaging the 3rd failure, which could be a real bug in alpha clusters
I've root caused csi-hostpath failure issue. It just failed to set the node properly in test code side. I will make a PR to fix it.
Or should it be done in your PR too, to avoid modifying twice in the separate PR? Actually, the fix are as below: (Failing to set node before pod creation)
Created for csi-hostpath fix, to prepare for the case that we need a separate PR. We can discuss more on this there, if needed.
/unassign found possible root cause for GCE issue: CSI NodeID annotation is not present on the Node object, could be caused by failed retries on updating Node object.

(apparently when alpha features are open we shouldn't be looking at the Node object anyway except we seem to be, so that might also be a bug). has the most context for the fix /assign
The symptom of the remaining test failure is the test pod fails to create because it fails to attach with the error: . The test succeeds the first time it's run in a cluster, and always fails after that. Two issues caused this error message. the CSI external-attacher tries to fetch the object to get the NodeID, it does so through an informer. The first time the test is run, objects are created for the very first time, and the informer returns the correct object. In subsequent test runs, s from the first run is kept, and for some reason the informer fails to detect the object and returns empty. Making the API call directly works, I'll patch the external-provisioner with this to mitigate the issue, and this is sufficient to get the test to pass. The informer bug will need to be eventually fixed. fails to post the NodeID annotation sometimes, likely due to node status patch not being retried correctly. This is crucial for clusters without feature flag set, and this is the default for 1.13. Will add the appropriate retry mechanism.
Opened to track the informer issue
Are and potential PR for the only 2 fixes being actively worked on for this issue?
Only is needed. We have a workaround for
will you have a followup PR to enable the CSI alpha tests back? or is that already in flight?
also reenables the tests
OK with merged we should see the CSI alpha tests re-enable. to keep an eye out for CI signal
The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! /close
Closing this issue. <detailsIn response to : The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The tests are actually running (and passing) in the normal suite now, now that the raw block feature moved to beta.


Question: dc.sync(ctx, pdbName) ps.VerifyDisruptionAllowed(t, pdbName, 0) <del> verifyUnsupportedControllerEventEmitted(t, dc) </del> <ins> verifyEventEmitted(t, dc, "UnmanagedPods") </ins> }

When using PDBs with or (percentage), it is not supported to reference pods via a PDB selector, that do not have a supported controller owner reference: Deployment, ReplicationController, ReplicaSet, and StatefulSet controllers or a resource implementing subresource This is also documented Nevertheless, there are still people who create PDBs a pods without controllers that ignore this. For example PDBs that set , want to block eviction, and node drain (it works for them but the PDB status is invalid). Unfortunately there is no feedback for the user to receive when this happens. The KCM logs is the only place that informs the user when unmanaged pods are found. Which I think is insufficient. There should be some way to inform users about this. Since it cannot be know on admission we probably have to do it when reconciling the PDB and unmanaged pods are found. Options: an event when this is detected - probably needs to be throttled in some way a condition in a PDB status that would be set and admins could monitor these to detect misbehaving PDBs create a PDB create unmanaged pod observe incorrect status because this configuration is unsupported No response <detailsNA </details<detailsNA </details<detailsNA </details<details</details<details</details<details</details
/sig apps
/triage accepted This was discussed during today's sig-apps, we've agreed to add a warning event right next to the log statement from the original description
I'll take this on.
/assign
/reopen I did some more testing for the unsupported controllers and found out that these are covered by the event, so I posted a followup to reflect that The new event should cover only the naked pods
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: latestLease *coordinationv1.Lease // newLeasePostProcessFunc allows customizing a lease object (e.g. setting OwnerReference) <del> // before every time the lease is created/refreshed(updated). Note that an error will block // a lease CREATE, causing the controller to retry next time, but an error won't block a // lease UPDATE. </del> <ins> // before every time the lease is created/refreshed(updated). // Note that an error will block the lease operation. </ins> newLeasePostProcessFunc ProcessLeaseFunc }

Some external controller deleted node object while kubelet was alive Due to ignoring errors in kubelet's lease logic, it created lease without ownerReference set Kubelet doesn't create lease without ownerReference set in any case, even pathological one Some external controller deletes node object Garbage collector properly deletes associated lease object Kubelet tries to update object and it fails on Conflict (saying that UID doesn't match): We go to branch, call backoffEnsureLease and assign result to base, but backoffEnsureLease may return potentially invalid object if ensureLease returned error (which happens when newLeasePostProcessFunc returns error, e.g. when node doesn't exist). We The next loop calls c.newLease on the incorrect base and ignores error in line 181 (so the second attempt to call newLeasePostProcessFunc which also fails with an error) We do UPDATE with incomplete (without effect of newLeasePostProcessFunc) object Given that leases allow CreateOnUpdate a new object, without ownerReference is created: No response <details21.10 </details<details</details<details</details<details</details<details</details<details</details
/sig node
/assign
/cc
/triage accepted need to understand what is the right behavior here
We are in trouble for this problem too. I think It’s exactly as said. Lease has the special update endpoint which can create the new lease objects not only update them. If it receives a request with a new lease object which has non-nil UID field, returns a conflict error. This situation can be occured in the function when a lease object is deleted. The lease object is deleted when the node resource is deleted in the result of GC. When received a conflict error in updating a lease object and the parent node object doesn't exist, , and continue next loop. In the next loop, the function returns the new lease object without ownerReference and UID. As a result, it is created unexpectively by the special update endpoint.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten