Question: fp := NewFakeProxier(ipt, ipvs, ipset, nil) svcIP := "10.20.30.41" svcPort := 80 <del> svcExternalIPs := sets.NewString("50.60.70.81", "2012::51") </del> <ins> svcExternalIPs := sets.NewString("50.60.70.81", "2012::51", "127.0.0.1") </ins> svcPortName := proxy.ServicePortName{ NamespacedName: makeNSN("ns1", "svc1"), Port:           "p80",

Is this a BUG REPORT or FEATURE REQUEST? /kind bug What happened: In ipvs proxy mode, most service functions like "nodeport/loadbalance" is realized by the association of iptables/ipset/ipvs, yet most ipvs proxier ut only verified ipvs rules, which can't ensure service is functional. What you expected to happen: Check both ipvs rules and iptables chains/rules, ipset entries in ipvsproxier ut.
/sig network /area ipvs /assign
What's the current UT coverage number?
Current is 50.2%.
So, can we achieve 70%?
Yeah, we can do that, and that's definitely necessary.


Question: } handler := proxy.NewUpgradeAwareHandler(location, proxyRoundTripper, true, upgrade, &responder{w: w}) <ins> handler.InterceptRedirects = utilfeature.DefaultFeatureGate.Enabled(genericfeatures.StreamingProxyRedirects) handler.RequireSameHostRedirects = utilfeature.DefaultFeatureGate.Enabled(genericfeatures.ValidateProxyRedirects) </ins> handler.ServeHTTP(w, newReq) }

What happened: upgrade requests to aggregated APIs (like websocket-based watch requests) on a kube-apiserver using --ssh-user hang What you expected to happen: Websocket-based watch requests work properly How to reproduce it (as minimally and precisely as possible): a kube-apiserver configured to use a node dialer to reach services/pods (e.g. with ) an aggregated API server (like the sample-apiserver) on a node other than the kube-apiserver a websocket-based watch against the aggregated API Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery /cc lavalamp
The code in question is here: handlingInfo.proxyRoundTripper is built here: and inherits the proxyDialerFn built in from the following construction chain: However, drops the custom dialer contained inside and from the returned round tripper. That means that upgrade requests fall back to use the default dialer, which cannot reach the aggregated API server
cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/assign would be good to resolve with the apiserver proxy work


Question: - /srv/reactor/highstate-new.sls EOF <del> # Install Salt # # We specify -X to avoid a race condition that can cause minion failure to # install.  See https://github.com/saltstack/salt-bootstrap/issues/270 # # -M installs the master set +x curl -L --connect-timeout 20 --retry 6 --retry-delay 10 https://bootstrap.saltstack.com | sh -s -- -M -X set -x </del> <ins> install-salt master service salt-master start service salt-minion start </ins>

I saw this error in on the master after a failed cluster start: I'm looking into this, but I want to create a tracking issue as I think some other people may be hitting this.
It's trying to bring up and failing: The weird thing is that neither that file nor the file have been modified that recently:
Yes, to me it looks like salt just isn't finding the plugin. The last change to _states did add , but I've seen this succeed recently as well (http://kubernetes-). I'm going to try this a few times, to try to figure out whether it is failing every time or not. My guess is that it is failing semi-randomly, so it isn't related to the changes since last-fail.
Aha... this is logged at debug level from I'm guessing it's a salt version mismatch... investigating.
I guess AWS needs to lock salt versions like GCE does:
Confirmed that this is because of upstream salt breakage: - deprecated (?) Not sure what we should do here. I can lock to the same version as GCE, but if we find a bug in that version of salt then we can't update without breaking this script.
We don't write bugs. ;)
Can you just put this in an AWS-specific block? It seems like that should work.
(And by "this" I mean "future Salt variable gunk")
I think I'm going to try to lock AWS to the same version as GCE, and then think more about a long-term plan once AWS is working again!
from the looks of things I think vagrant will have the same problem (?) (Note that I am in the process of developing / testing the branch above, so I don't think you can just copy-and-paste the fix - yet!)
As a short term fix, using 2015.05 (ubuntu trusty salt ppa), was to change from: import as ipaddr to: import as ipaddr this actually seems to be working. might be deprecated. but not in 2015.05. why is there a containerbridge custom state at all btw? why not use network.managed and
- I'm not sure we can just change the import, because I think we have to be compatible with GCE's version of salt also.

I'm not sure of why we did it this way vs other ways of doing things in salt
Does this mean that the official k8s distribution is broken on AWS, BTW? When is the next point release (and where are these discussed)?
We cherry-pick into point releases as necessary, and it's been pretty informal and usually initiated by us (Google). I'd actually be happy to have a discussion about how to initiate it externally (we're going to need a policy for requesting such things). cc
FWIW, policy aside, I'm happy to cut a point release for this once you think is tested well enough.
That would be great, thanks. I'll be testing it pretty hard today, I think. It has occurred to me that other clouds may also be broken. By grepping, it looks like vagrant and vsphere use I am utterly unfamiliar with the vsphere implementation, and I think vagrant sidesteps the issue because on Redhat is skips the call to It might be good to get confirmation that vagrant hasn't fallen in to a different trap though! Let me test it out today (unless people start filing lots of issues, in which case I think we roll it out more urgently...)
I will test out the Vagrant setup today, but I think we should be good because we do not use the same networking setup.
File a bug for a 0.17.1 to get this cherry-picked and assign it to me, as a reminder?
Do you know the status of 17.1? I'm seeing the same problems trying to AWS working, even tried an older release (e.g. v15), and have the same problem. My script looks 32 times on "[salt-master not working yet]" then gives up.
We're cutting one today.
I'm still seeing the same issue as before with the new release, my master starts then times out waiting for salt-master to start. Do you know some spots I can look at to help troubleshoot? Do you want a new issue created to track against?
Please file a new issue and tag thanks. I'm sorry it didn't fix things for you! On May 22, 2015 4:49 AM, "Steve Sloka" wrote:


Question: }, }, update: func(job *batch.Job) { <del> // change something. </del> job.Annotations["foo"] = "bar" }, }, <ins> "deleting user annotation": { job: &batch.Job{ ObjectMeta: metav1.ObjectMeta{ Name:            "myjob", Namespace:       metav1.NamespaceDefault, ResourceVersion: "0", Annotations: map[string]string{ batch.JobTrackingFinalizer: "", "foo":                      "bar", }, }, Spec: batch.JobSpec{ Selector:       validSelector, Template:       validPodTemplateSpec, ManualSelector: pointer.BoolPtr(true), Parallelism:    pointer.Int32Ptr(1), }, }, update: func(job *batch.Job) { delete(job.Annotations, "foo") }, }, </ins> "updating node selector for unsuspended job disallowed": { job: &batch.Job{ ObjectMeta: metav1.ObjectMeta{

The gets very long testing mulitple functions in such as , and . These checks should be performed in dedicated Test functions using table for different scenarios. Target functions and are being as part of The remaining checks should be migrated to these functions. Also, a test function should be to migrate the remaining tests.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Any progress on this? I'm working on and I noticed that has relatively low coverage so I was going to work on getting the coverage up.
I created that is somewhat related to this.
no progess. go ahead please
/assign


Question: FrameworkExtensionPointDuration, SchedulerQueueIncomingPods, SchedulerGoroutines, <ins> PermitWaitDuration, </ins> } )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : metric name Description -- -- permitwaitduration_seconds Duration of permit extension point finishes waiting. Why is this needed: In permit extension point, if the status is "Wait", we block until the pod is scheduled or timeout: Blocking is an important behavior that we should monitor. /sig scheduling
/assign


Question: "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/api/storage/v1:go_default_library", "//staging/src/k8s.io/api/storage/v1beta1:go_default_library", <del> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", </del> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/fields:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: This switches NodeLister to scheduler cache for . However, the new implementation in the cache never returns IsNotFound error any more. Rather, it returns a normal error: However, we rely on the error type for logging, for example: What you expected to happen: The cache should return IsNotFound error if applicable. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig scheduling cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi. I'd like to work on this for #Hacktoberfest, and for my very first open-source contribution.
Hello, I would love to contribute too in case. I'm already looking at the code in fact :D
Thanks all for the interest, will assign to since he expressed interest first. /assign
I was browsing around the rest of the implementation of the cache and noticed that none of the other accessors return . Also, I couldn't determine if ever returned anything other than an . I'm new here, so I may be doing it wrong. Are you sure the behavior changed, and are you sure returning is expected? Should the other accessors be changed, too?
Let's look at previous release. The object is initialized in [] where it creates a .

Now let's take a look at implementation: There you can see that it calls NodeLister Get() method, which can return a IsNotFound error.
Hello, is this issue being solved ? If not, i am interested in solving this issue.
Thanks but I am working on this currently.
, if you haven't started your work yet, let me know if I can raise a PR for this issue. I have the change ready.
I started, but haven’t been able to finish creating the PR. Go ahead and submit yours. Thanks. Sent from my iPhone


Question: return unmountedVolumes } <del> func (asw *actualStateOfWorld) SyncReconstructedVolume(volumeName v1.UniqueVolumeName, podName volumetypes.UniquePodName, outerVolumeSpecName string) { asw.Lock() defer asw.Unlock() if volumeObj, volumeExists := asw.attachedVolumes[volumeName]; volumeExists { if podObj, podExists := volumeObj.mountedPods[podName]; podExists { if podObj.outerVolumeSpecName != outerVolumeSpecName { podObj.outerVolumeSpecName = outerVolumeSpecName asw.attachedVolumes[volumeName].mountedPods[podName] = podObj } } } } </del> func (asw *actualStateOfWorld) newAttachedVolume( attachedVolume *attachedVolume) AttachedVolume { seLinuxMountContext := ""

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We have even encountered such case that pod continuously reports “Unable to attach or mount volumes for pod;...” when kubelet restarted. The reasons are listed in details as below: , as result of apiserver instantaneous unstablility, dswp get pvc‘s pv spec failed, then the pvc volume does not be to dsw. , syncStates() lists all the volume directories locally, and recontructs the volume from local volume, if the volume isn't in dsw, it will put the recontructed volume to asw, but now the volume.OuterVolumeSpecName isn't correct. chance, at this time, In , the fail to pvc volume is to dsw again normally in reconcile() -mountAttachVolumes(), it compares dsw and asw by volume's innerVolumename, it's all the same. but it could not sense the OuterVolumeSpecName dismatch. so it will not re-mount again to update asw cache , it checks pod's volume attach or mount success by comparing volume.OuterVolumeSpecName and , but this pv'c OuterVolumeSpecName isn't correct in asw. so pod continuously reports “Unable to attach or mount volumes for pod;...” reconcile() could sense the reconstruct volume's dismatch to the dsw, then re-mount again to make volume cache consistency Kubernetes version (use ): 1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
CC
I think this is a point that can be optimized. /cc
I will give a fix for this
Thanks for reporting the issue, this innerSpecName and outerSpecName are confusing and cause issues. Please ping me for review when you have a fix. Thanks!
Please don't triage your own issues immediately, give someone else a chance to review them :( /sig storage
okay, I know


Question: allowSpecificLocalhost = map[string]string{ AllowedProfilesAnnotationKey: v1.SeccompLocalhostProfileNamePrefix + "foo", } <ins> allowSpecificDockerDefault = map[string]string{ AllowedProfilesAnnotationKey: v1.DeprecatedSeccompProfileDockerDefault, } allowSpecificRuntimeDefault = map[string]string{ AllowedProfilesAnnotationKey: v1.SeccompProfileRuntimeDefault, } </ins> ) func TestNewStrategy(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In k8s 1.19.2 If the seccomp annotations of psp doesn't contain , pod with seccomp annotations would not start with error: psp with works psp without doesn't work Is this due to ? What you expected to happen: In k8s 1.18, psp annotations without works. I think in k8s 1.19, it should also work.
/sig auth
/assign I think this was a regression with the seccomp graduation to GA. Since we auto-convert 'docker/default' to the runtime default in fields, but PSPs that only allow 'docker/default' aren't accepting 'runtime/default'. I think the easiest fix is to just add both and to the allowed profiles map whenever either is set (here: ) This will need to be cherry-picked /priority important-soon
I'm happy to work on this one tomorrow if is not tackling the issue earlier :upsidedownface:
Fix is incoming in
Thanks for the quick fix ! Can you open a cherry-pick for v1.19 too? (see )
Forgot to link it, but this is already done: :)
Is there an easy way to check in which version of k8s this fix is applied? Label, milestone, etc? Another question, what might be a workaround without updating cluster?
Find the fix commit and run And there's also a cherry-pick commit: The workaround could be add annotations to your PSPs, take a look at this PR:


Question: "time" v1 "k8s.io/api/core/v1" <ins> apiequality "k8s.io/apimachinery/pkg/api/equality" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/diff" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/kubernetes/test/e2e/framework" "k8s.io/kubernetes/test/e2e/instrumentation/common"

[X] test approval issue : [X] test pr : [x] two weeks soak end date : 15 Jul 2022 [x] test promotion pr : According to this APIsnoop query, there are still a single remaining Event endpoint which is untested. replaceCoreV1NamespacedEvent Using a number of existing e2e test practices a new has been created for this pod status endpoint. The e2e logs for this test are listed below. This query shows the event endpoints hit within a short period of running the e2e test. If a test with these calls gets merged, test coverage will go up by 1 point. This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // MarkAllPodsNotReady updates ready status of all pods running on // given node from master return true if success func MarkAllPodsNotReady(kubeClient clientset.Interface, node *v1.Node) error { <del> // Don't set pods to NotReady if the kubelet is running a version that // doesn't understand how to correct readiness. // TODO: Remove this check when we no longer guarantee backward compatibility // with node versions < 1.2.0. if NodeRunningOutdatedKubelet(node) { return nil } </del> nodeName := node.Name glog.V(2).Infof("Update ready status of pods on node [%v]", nodeName) opts := metav1.ListOptions{FieldSelector: fields.OneTermEqualSelector(api.PodHostField, nodeName).String()}

As discussed on slack with we might want to clean the node controller from code that ensures backward compatibility for version 1.4 and earlier ones. /sig node
/assign
- how do we plan to devide this one ? The following order for merge should be fine I think: the tests (we can split also test case deletion if it's too long to review): logic itself + kubeletVersion on the according tests: change how we create nodecontroller: clean consts and BUILD:
Yeah, sounds good.
I'll create the PRs accordingly and assign them to you


Question: testDynamicProvisioning(test, c, claim, nil) }) }) <ins> Describe("Invalid AWS KMS key", func() { It("should report an error and create no PV", func() { framework.SkipUnlessProviderIs("aws") test := storageClassTest{ name:        "AWS EBS with invalid KMS key", provisioner: "kubernetes.io/aws-ebs", claimSize:   "2Gi", parameters:  map[string]string{"kmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/55555555-5555-5555-5555-555555555555"}, } By("creating a StorageClass") suffix := fmt.Sprintf("invalid-aws") class := newStorageClass(test, ns, suffix) class, err := c.StorageV1().StorageClasses().Create(class) Expect(err).NotTo(HaveOccurred()) defer func() { framework.Logf("deleting storage class %s", class.Name) framework.ExpectNoError(c.StorageV1().StorageClasses().Delete(class.Name, nil)) }() By("creating a claim object with a suffix for gluster dynamic provisioner") claim := newClaim(test, ns, suffix) claim.Spec.StorageClassName = &class.Name claim, err = c.CoreV1().PersistentVolumeClaims(claim.Namespace).Create(claim) Expect(err).NotTo(HaveOccurred()) defer func() { framework.Logf("deleting claim %q/%q", claim.Namespace, claim.Name) err = c.CoreV1().PersistentVolumeClaims(claim.Namespace).Delete(claim.Name, nil) if err != nil && !apierrs.IsNotFound(err) { framework.Failf("Error deleting claim %q. Error: %v", claim.Name, err) } }() // Watch events until the message about invalid key appears err = wait.Poll(time.Second, framework.ClaimProvisionTimeout, func() (bool, error) { events, err := c.CoreV1().Events(claim.Namespace).List(metav1.ListOptions{}) Expect(err).NotTo(HaveOccurred()) for _, event := range events.Items { if strings.Contains(event.Message, "failed to create encrypted volume: the volume disappeared after creation, most likely due to inaccessible KMS encryption key") { return true, nil } } return false, nil }) Expect(err).NotTo(HaveOccurred()) }) }) </ins> }) func getDefaultStorageClassName(c clientset.Interface) string {

This is continuation of and . What happened: When StorageClass refers to a key that is not accessible to Kubernetes (i.e. AWS API returns AccessDeniedException), AWS plugin still tries to provision a volume under assumption that while Kubernetes can't access the key, the other components may access it and create the volume and attach it to a node later. Related code is - checks only for . Other errors are just logged and provisioning continues. Problem is that when AWS really can't access the key, it will provision some volume (ec2.CreateVolume returns success + valid AWS EBS), but this AWS EBS disappears after few seconds. Therefore Kubernetes does not know that something went wrong and creates a PV for the returned EBS. Kubernetes later fails to run a pod that refers to such PV, because the PV does not exist. What you expected to happen: Kubernetes must not create PV in this case and must report error during provisioning of such volume. Ideas: According to , the only signal that AWS produces is an event in Amazon CloudWatch Events that a key could not be used for encryption. Kubernetes AWS cloud provider is not subscribed to these events and does not get this notification. We would need new permission for controller-manager to get the events and some new code to filter them and report back to user. Kubernetes AWS cloud provider could wait for X seconds and re-check that created volume still exists and did not disappear because of inaccessible key. What would be reliable value of X? Kubernetes volume plugin for AWS could return both a warning "I failed to check encryption key, created volume may not work" that would be propagated to user and the suspicious volume at the same time, but the API between volume plugins and Kubernetes allows only one - either error or volume. It could be extended, still the user experience would be suboptimal. Kubernetes could send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions. Again, UX is quite bad here. /kind bug /sig aws
Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. If the main source of errors are sent via CloudWatch Events, then best practice would seem to be to subscribe via SNS to receive those events.

Possibly fall back to vague "send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions," where a SNS topic isn't provided, but otherwise use CloudWatch events to enrich state/messages. Would then also need to change docs to reflect needing to set up CloudWatch Events -SNS topic. Big drawback of going down this route is a fair few downstream cluster bootstrap projects would likely need updating to support it in their CloudFormation / Terraform bits.
According to it takes quite a long time until an event is available in CloudWatch events. what was the average / maximum delay?
I haven't calculated the metrics accurately but the cloudwatch events arrival vary(and standard deviation was high). Using cloudwatch events was also considered for maintaining instance cache and as well concluded that - they are not realtime (AWS itself documents them as "near realtime"). It may be - we need to measure them again and find out if we can use them for the purpose here. Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. from jan's analysis above it appears that - if IAM does not have permission to DescribeKey then EBS volume gets created but then it disappears soon. So net result still is - if "current user" can't describekey, she can't provision the volume. Is there any documentation that indicates otherwise? cc
Testing this for real now. User creates volume with no access to underlying CMK, or insufficient permissions does not seem to generate events with sufficient information as to what went wrong, i.e. don't appear to receive events for create volume failure, only deletion. E.g.: This policy is sufficient to allow the creation of volumes - isn't necessary for the volume to be created. It isn't sufficient to attach the volume however, creating the following error, The permission's required to fix this, to delegate access to EC2 itself: Bit of experimentation shows that only is required for attaching - the ones for creating the volume aren't needed at the attachment stage. The docs at is therefore correct for volume creation, but doesn't describe the volume attachment scenario.
So, what is our solution then? From the above it seems Kubernetes can't reliably check if a key is usable for encryption of a PV.

And it can't get an usable event either.
Possibly one of the following: change. Live with the pain. "GenerateDataKey" and "Decrypt" on the generated ciphertext blob prior to volume creation as a preflight check. The advantage being that we can throw an error that this isn't going to work beforehand, but with the disadvantage of doubling up API calls. This is probably fine, and I think if you're creating encrypted volumes at a rate that's likely to run into API call limits or get super expensive, you might have other problems? We then also do the CreateGrant check during volume attachment (as an optional extra). the same check, but only after an error. Problem being that we get much less in the way of guaranteeing what the root cause is, but we don't add to the API calls in the happy path. In any case, is this bug biting sufficiently to fix here, or should we start thinking about the CSI implementation?
thanks a lot for detailed investigation. I'll try to get some time to implement 2. even before CSI. Users are very innovative in setting up their cluster permissions wrong and the messaging to user is very confusing in this particular issue.
/assign
2. does not work. Kubernetes does not need to have GenerateDataKey and Decrypt permissions to create encrypted volume. My AWS account is severely limited (I am using shared company account) and I can't call GenerateDataKey on a key: But I can create encrypted EBS using the same key: The volume gets created successfully. It's not the user who runs Kubernetes who needs permissions for GenerateDataKey, it's AWS itself.
And the volume stayed created and didn't get reaped after a few seconds? I was confused about this part, it didn't seem to follow how the APIs should work. Has anyone ever created a volume with that key, or is it a new one? I'll try what I did previously in a bit across a few regions.
Yes, the volume stayed around for several minutes, then I deleted it manually. The key is our default encryption key, so when I create an encrypted volume with no key, I get this one. No zones, no regions involved.
I got an idea, the volume is "creating" and then it gets "available".

I could in a loop until newly created volume either gets "Available" or deleted. This will abuse the API quota ( is cheap, right?) + it will slow down provisioning a bit - all volumes I tried were Available in a second or two. Will it work reliably enough?
Might be the difference then, as I don't think the same perms as I described above are needed for the default key as I assume that it gets automatically delegated to the EC2 service. I only tested & documented the explicit, alternate key path above. Your approach should work, but I don't know when EC2 API limits will kick in and how soon users would see that happen. Worth a shot though?
It takes ~4 seconds for a volume to reach "available", it's quite a long time. This will affect encrypted volumes with explicit keys only.
Fix:


Question: "github.com/davecgh/go-spew/spew" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/client-go/informers" coreinformers "k8s.io/client-go/informers/core/v1" "k8s.io/client-go/kubernetes/fake"

What happened: kube-controller-manager with enabled option --controllers=,tokencleaner is not removing newly created tokens. It works only after kube-controller-manager restart or after editing secret with token. What you expected to happen: It should remove all expired tokens from kube-system namespace, without restarting or touching secrets. How to reproduce it (as minimally and precisely as possible): enable tokencleaner in kube-controller-manager start controller manager create bootstrap token, you can use script wait 5 minutes token is still there restart kube-controller-manager OR edit newly created token token should be gone after a while Anything else we need to know?: It could be related with Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Do you see the following log ? thanks
no errors at all
/sig auth


Question: <del> #cloud-config write_files: - path: /etc/cloud.conf permissions: 0600 content: | [Global] auth-url = OS_AUTH_URL username = OS_USERNAME api-key = OS_PASSWORD tenant-id = OS_TENANT_NAME region = OS_REGION_NAME [LoadBalancer] subnet-id = 11111111-1111-1111-1111-111111111111 - path: /opt/bin/git-kubernetes-nginx.sh permissions: 0755 content: | #!/bin/bash git clone https://github.com/thommay/kubernetes_nginx /opt/kubernetes_nginx /usr/bin/cp /opt/.kubernetes_auth /opt/kubernetes_nginx/.kubernetes_auth /opt/kubernetes_nginx/git-kubernetes-nginx.sh - path: /opt/bin/download-release.sh permissions: 0755 content: | #!/bin/bash # This temp URL is only good for the length of time specified at cluster creation time. # Afterward, it will result in a 403. OBJECT_URL="CLOUD_FILES_URL" if [ ! -s /opt/kubernetes.tar.gz ] then echo "Downloading release ($OBJECT_URL)" wget "${OBJECT_URL}" -O /opt/kubernetes.tar.gz echo "Unpacking release" rm -rf /opt/kubernetes || false tar xzf /opt/kubernetes.tar.gz -C /opt/ else echo "kubernetes release found. Skipping download." fi - path: /opt/.kubernetes_auth permissions: 0600 content: | KUBE_USER:KUBE_PASSWORD coreos: etcd2: discovery: https://discovery.etcd.io/DISCOVERY_ID advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001 initial-advertise-peer-urls: http://$private_ipv4:2380 listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001 listen-peer-urls: http://$private_ipv4:2380,http://$private_ipv4:7001 flannel: ip_masq: true interface: eth2 fleet: public-ip: $private_ipv4 metadata: kubernetes_role=master update: reboot-strategy: off units: - name: etcd2.service command: start - name: fleet.service command: start - name: flanneld.service drop-ins: - name: 50-flannel.conf content: | [Unit] Requires=etcd2.service After=etcd2.service [Service] ExecStartPre=-/usr/bin/etcdctl mk /coreos.com/network/config '{"Network":"KUBE_NETWORK", "Backend": {"Type": "host-gw"}}' command: start - name: generate-serviceaccount-key.service command: start content: | [Unit] Description=Generate service-account key file [Service] ExecStartPre=-/usr/bin/mkdir -p /var/run/kubernetes/ ExecStart=/bin/openssl genrsa -out /var/run/kubernetes/kube-serviceaccount.key 2048 2>/dev/null RemainAfterExit=yes Type=oneshot - name: docker.service command: start drop-ins: - name: 51-docker-mirror.conf content: | [Unit] # making sure that flanneld finished startup, otherwise containers # won't land in flannel's network... Requires=flanneld.service After=flanneld.service Restart=Always - name: download-release.service command: start content: | [Unit] Description=Downloads Kubernetes Release After=network-online.target Requires=network-online.target [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/bin/bash /opt/bin/download-release.sh - name: kube-apiserver.service command: start content: | [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network-online.target Requires=network-online.target After=download-release.service Requires=download-release.service Requires=generate-serviceaccount-key.service After=generate-serviceaccount-key.service [Service] ExecStartPre=/usr/bin/ln -sf /opt/kubernetes/server/bin/kube-apiserver /opt/bin/kube-apiserver ExecStartPre=/usr/bin/mkdir -p /var/lib/kube-apiserver ExecStart=/opt/bin/kube-apiserver  --address=127.0.0.1  --cloud-provider=rackspace  --cloud-config=/etc/cloud.conf  --etcd-servers=http://127.0.0.1:4001  --logtostderr=true  --port=8080  --service-cluster-ip-range=SERVICE_CLUSTER_IP_RANGE  --token-auth-file=/var/lib/kube-apiserver/known_tokens.csv  --v=2  --service-account-key-file=/var/run/kubernetes/kube-serviceaccount.key  --service-account-lookup=true  --admission-control=NamespaceLifecycle,NamespaceAutoProvision,LimitRanger,SecurityContextDeny,ServiceAccount,DefaultTolerationSeconds,ResourceQuota Restart=always RestartSec=5 - name: apiserver-advertiser.service command: start content: | [Unit] Description=Kubernetes Apiserver Advertiser After=etcd2.service Requires=etcd2.service After=master-apiserver.service [Service] ExecStart=/bin/sh -c 'etcdctl set /corekube/apiservers/$public_ipv4 $public_ipv4' Restart=always RestartSec=120 - name: kube-controller-manager.service command: start content: | [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes After=network-online.target Requires=network-online.target After=kube-apiserver.service Requires=kube-apiserver.service [Service] ExecStartPre=/usr/bin/ln -sf /opt/kubernetes/server/bin/kube-controller-manager /opt/bin/kube-controller-manager ExecStart=/opt/bin/kube-controller-manager  --cloud-provider=rackspace  --cloud-config=/etc/cloud.conf  --logtostderr=true  --master=127.0.0.1:8080  --v=2  --service-account-private-key-file=/var/run/kubernetes/kube-serviceaccount.key  --root-ca-file=/run/kubernetes/apiserver.crt Restart=always RestartSec=5 - name: kube-scheduler.service command: start content: | [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes After=network-online.target Requires=network-online.target After=kube-apiserver.service Requires=kube-apiserver.service [Service] ExecStartPre=/usr/bin/ln -sf /opt/kubernetes/server/bin/kube-scheduler /opt/bin/kube-scheduler ExecStart=/opt/bin/kube-scheduler  --logtostderr=true  --master=127.0.0.1:8080 Restart=always RestartSec=5 #Running nginx service with --net="host" is a necessary evil until running all k8s services in docker. - name: kubernetes-nginx.service command: start content: | [Unit] Description=Kubernetes Nginx Service After=network-online.target Requires=network-online.target After=docker.service Requires=docker.service [Service] ExecStartPre=/opt/bin/git-kubernetes-nginx.sh ExecStartPre=-/usr/bin/docker rm kubernetes_nginx ExecStart=/usr/bin/docker run --rm --net="host" -p "443:443" -t --name "kubernetes_nginx" kubernetes_nginx ExecStop=/usr/bin/docker stop kubernetes_nginx Restart=always RestartSec=15 </del>

Rackspace support has not ben updated in a few months. cluster/rackspace/cloud-config/minion-cloud- is using etcd_servers for kubelet and kube-proxy, but support for them has been removed since v0.9, I think. I'll remove it in a few days if no-one makes a counter-offer to get the config working with latest release.
There's been some work on this but it is blocked while we figure out some load-balancer design stuff. , Eric Tune wrote:
Okay, I will remove the link to rackspace until it is fixed (). Leaving this issue open to remind us to delete it in, say, a month, if no progress is made.
cc
See also
I'd prefer to try to get things updated and keep support. However I don't want to continually become stale. every few months either.
Hi all, I want to use kubernetes on my existing rackspace infrastructure but I don't understand if rackspace suppor was removed or not. In Doc, under "custom cloud solution" it's linked but I wasn't able to deploy it neither with install method nor with . With i got a error (I have 16GB or RAM!). thanks in advance, Fx
It looks like no support has happened on Rackspace for a year.
You should not expect it to work, and we should delete rackspace support.
ACK.
Unfortunately I think we'll need to drop support for Rackspace as a deployment target since I can't commit to keeping the deployment scripts up to date. Rackspace support as a cloudprovider should be maintained.
I would be willing to support the Rackspace deployment target, if there is interest to keep it alive.
There would be definitely an interest from our side - sad to see that Rackspace has no interest in maintaining it
No problem, I will take over the maintenance. It will take me a bit of time to get caught up, but I will get it done as soon as possible.
Do you think it would be hard to get the of LBaaS working ?
Sorry it took me a bit to respond. there is a quick update as to where I am. I was busy with some stuff, that prevented me from working on this. But that has been resolved, and should have updates by the end of next week.

However what's causing the delays is the lack of a good API for Rackspace Cloud. I've used the gophercloud, it works but I'm not a fan. Additionally gophercloud doesn't have full API support (one of the things I'm not a fan). This would make it hard to implement LBaaS. I should have a better idea of your request sometime next week.
I have seen this api docu: - is it the same as you saw? I think it should be doable with it.


Question: set = obj.(*apps.StatefulSet) } if set.Status.Replicas != 0 { <del> t.Error("Falied to scale statefulset to 3 replicas") </del> <ins> t.Errorf("set.Status.Replicas = %v; want 0", set.Status.Replicas) </ins> } }

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen


Question: t.Errorf("Unexpected blocking pet, err %v: %+v", err, p) } } <ins> // TODO(mkwiek): test if the petset.Status.Replicas is actually correct func TestPetSetReplicaCount(t *testing.T) { psc, fc := newFakePetSetController() ps := newPetSet(3) i, _ := psc.syncPetSet(ps, fc.getPodList()) if i != len(fc.getPodList()) { t.Errorf("syncPetSet should return actual amount of pods") } } </ins>

Is this a request for help? Nope What keywords did you search in Kubernetes issues before filing this one? petset replicas current desired Is this a BUG REPORT or FEATURE REQUEST? BUG Kubernetes version Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-master+", GitCommit:"", GitTreeState:"not a git tree", BuildDate:"2016-09-01T10:49:36Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-master+", GitCommit:"", GitTreeState:"not a git tree", BuildDate:"2016-09-01T10:49:36Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: 24 cores, 64 GB RAM OS Arch Linux Kernel (e.g. ): Linux hostname 4.4.13-1-lts SMP Wed Jun 8 16:44:31 CEST 2016 x86_64 GNU/Linux Install tools: local-cluster- Others: What happened: While creating PetSet based on this tutorial: first PetSet pod was stuck in pending state due to lack of PersistentVolumes in cluster. When getting the petset info via I expected to see "CURRENT" set to 1, while the kubectl output was as follows: While clearly shows that only one pod is created: In json and yaml output, status.replicas value was also 2. What you expected to happen: I expected that CURRENT value shown by kubectl would be due to only one pod being created. How to reproduce it (as minimally and precisely as possible): Create via on empty cluster and run and . Anything else do we need to know: Nope
CURRENT is the number of pods that have been created, not that they are READY. See for example the latest api change for replica sets where we exactly for this reason. All controllers (petsets, daemonsets, deployments, etc.) should have ReadyReplicas cc:
See also
I guess it doesn't work anyway, since only one pod was created for this PetSet, but the CURRENT says that 2 were created. I will change bug description then.

Hmm, the idea is to sync every pet periodically, but only create them one at a time. The logic that counts status.replicas is wrong. This counter shouldn't be incremented in the case where we didn't update/sync or create a pet: Today we do increment it all pet creation is blocked on the first pet becoming unhealthy. Should be an easy fix if anyone wants to jump in ( fyi) otherwise I'll get to it once 1.4 is out.
I can try to fix it if that's OK. Thanks for the pointer to the source code!
Please do, co-assigned


Question: } // Update the revisions name and labels clone.Name = ControllerRevisionName(parent.GetName(), hash) <del> created, err := rh.client.Apps().ControllerRevisions(parent.GetNamespace()).Create(clone) </del> <ins> created, err := rh.client.AppsV1beta1().ControllerRevisions(parent.GetNamespace()).Create(clone) </ins> if errors.IsAlreadyExists(err) { probe++ continue

The statefulset controller and controller history implementations should be migrated off of the Apps client (deprecated) and onto the AppsV1BetaClient. Currently there is no user facing issue caused by the use of the deprecated resource, but we should maintain good code hygiene.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/assign kow3ns
/assign kow3ns
/cc
/assign


Question: err := cs.CoreV1().Pods(f.Namespace.Name).Delete(context.TODO(), podName, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete pod: %s in namespace: %s", podName, f.Namespace.Name) }() <del> ginkgo.By("dumping iptables rules on the node") </del> <ins> // wait until host port manager syncs rules </ins> cmd = "sudo iptables-save" <del> framework.Logf("Executing cmd %q on node %v", cmd, node.Name) result, err := hostExec.IssueCommandWithResult(cmd, node) if err != nil { framework.Failf("Interrogation of iptables rules failed on node %v", node.Name) </del> <ins> if framework.TestContext.ClusterIsIPv6() { cmd = "sudo ip6tables-save" </ins> } <ins> err = wait.PollImmediate(framework.Poll, framework.PollShortTimeout, func() (bool, error) { framework.Logf("Executing cmd %q on node %v", cmd, node.Name) result, err := hostExec.IssueCommandWithResult(cmd, node) if err != nil { framework.Logf("Interrogation of iptables rules failed on node %v", node.Name) return false, nil } </ins> <del> ginkgo.By("checking that iptables contains the necessary iptables rules") found := false for _, line := range strings.Split(result, "n") { if strings.Contains(line, "-p sctp") && strings.Contains(line, "--dport 5060") { found = true break </del> <ins> for _, line := range strings.Split(result, "n") { if strings.Contains(line, "-p sctp") && strings.Contains(line, "--dport 5060") { return true, nil } </ins> } <del> } if !found { </del> <ins> framework.Logf("retrying ... not hostport sctp iptables rules found on node %v", node.Name) return false, nil }) if err != nil { </ins> framework.Failf("iptables rules are not set for a pod with sctp hostport") } ginkgo.By("validating sctp module is still not loaded")

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: https://k8s- Which test(s) are failing: sig-network] SCTP [Feature:SCTP] [LinuxOnly] should create a Pod with SCTP HostPort Since when has it been failing: Test [sig-network] SCTP [Feature:SCTP] [LinuxOnly] should create a Pod with SCTP HostPort has failed 3 times since it last passed at commit/. It started failing between commit/ (Tue Jun 09 13:25 PDT) and commit/ (Wed Jun 10 01:26 PDT): Testgrid link: https://k8s- Reason for failure: Anything else we need to know: I'm not sure if we already did it but there was a proposal to change from dockershim to containerd for the GCE jobs
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
GitHub didn't allow me to assign the following users: janosi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yes, that would break us... the test checks "are we using kubenet", but it didn't occur to me that we would need to deal with kubenet with CRI instead of kubenet with dockershim. It should be easy to make it check for both the dockershim rule and the CNI portmap rule though...
yes, according to the "Changes" link on the dashboard, containerd replaced docker in GCE right before the failures.

Let's see if I can make it :)
So actually the test is pretty generic (it just looks for and ) and it seems like it ought to pass correctly with CNI portmap too. I'm not sure what's failing. You may need to add some debugging...
OK, so as I can see, containerd's CRI implementation just ignores all HostPort definitions that do not have TCP or UDP Protocol value. I.e. containerd will not send HostPort mapping definitions to the CNI if the protocol is not TCP or UDP.
seems this is the reason why sctp was skipped
Yes, exactly. Introduced right after SCTP support was merged in k8s. Just found this in the package: Though not sure it "knows" the right runtime at runtime, because there is no user of it in :)
just curious, why do we skip in containerd and allow it with dockershim?
Hmm, do you mean why do not we skip in case of containerd? The reason is that I did not remember that containerd PR about the SCTP support. Let's see if I can skip it.
Hm, that is not true. My search engine was wrong. After a restart I found usage of that in e2e_node
Has a bug been filed upstream in containerd? Just skipping the tests seems like the wrong approach here.
sorry, I meant why we are not allowing it in containerd, does this feature should work in all container runtimes? CRIO, docker and containerd? if affirmative, we can submit a PR to containerd/cri to allow SCTP, I can do it if you don't have time
Excuse me, I misunderstood your question. Yep, why not. I mean, it could work in all container runtimes. As I understand the containerd implementation is based on a misunderstanding: that the host port mapping loads the SCTP kernel module and that would be bad for those apps that want to use user space SCTP. But as long as there is no SCTP socket created on a node the SCTP kernel module is not loaded.
I can initiate a discussion with containerd about this.
seems this dates back to the original SCTP KEP, seems like it should be a pretty simple fix (?) thanks for filing the bug and for the details there!

CRI-O seems to have the same issue:
Just confirming you want this filter to be gone now. Or do you need the filter to be manageable in some way.. filtered by a config switch?
For TCP and UDP HostPorts, the runtime is expected to do two things: a listening socket on that port so the kernel will think it's in use and not allocate it to anyone else iptables rules (or whatever) to route traffic from the host port to the container port For SCTP HostPorts, the runtime should do the second part (iptables rules), but should not do the first part (listening socket), for two reasons: Some users want to use "userspace SCTP", in which case they will be referring to SCTP at the Kubernetes level but need to keep the kernel module from ever being loaded. (This is discussed more .) If is not loaded when Kubernetes starts, we want to avoid having Kubernetes itself cause it to be auto-loaded, for security reasons. (More discussion of this .) So, runtimes should be filtering out SCTP for the opening-listening-sockets part, but not for the making-iptables-rules part. (Some implementations, notably CNI , don't actually do the opening-listening-sockets part, so they wouldn't be filtering at all.)
Exactly. The current state in the runtimes: Docker (dockershim, actually): OK. The logic was implemented together with the basic SCTP feature, so listening on the host port is filtered for SCTP, but the iptables rules are created Containerd (CRI, actually): the patch which was applied 2 years ago filters out every SCTP host port handling, because it filters SCTP ports from the list of port mappings that are passed to CNI (portmap) later. Actually, containerd (CRI) does not perform the "listen" part at all, so the patch is not needed at all CRI-O: the patch applied 2 years ago filters out everything. CRI-O imports the host port manager implementation from kubelet/dockershim. For this reason it has the logic that can filter out SCTP ports from the "listen" part, but still applies the iptables rules. I.e. the patch is not needed in CRI-O either.
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale The test is flaky now :thinking: https://k8s-
I guess it is a timing issue? Do you think some extra polling rounds would help? (I did not disappear, but currently I am very loaded with my regular job...)
I've submitted a PR to fix it and another one to test the sctp connectivity, please review them, I've pinged you there :smile:


Question: if err != nil { return false, err } if response.Status.Allowed { fmt.Fprintln(o.Out, "yes") } else {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When using kubectl auth can-i <verb<resourcesthe "verb" isn't check and returns yes on any resource regardless of what the verb is. What you expected to happen: I expected the cli to register an error like when you don't have enough arguments. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): client version GitVersion: 1.19.2 server version: GitVersion: 1.19.0 Cloud provider or hardware configuration: Mac OS X OS (e.g: ): n/a Kernel (e.g. ): Darwin Install tools: brew Network plugin and version (if this is a network-related bug): n/a Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
This is working as intended. The authorization layer can answer questions about arbitrary verbs, and the response to can-i mirrors the response you would get from a subjectaccessreview permission check. Getting a "yes" response to arbitrary verbs likely means you are a superuser and would be authorized for any operation.
Wouldn't a superuser only have access to the Kubernetes verbs that exist and not the arbitrary verbs that it is allowed to query that is defined here () This seems to go against the selfsubjectaccessreview which will return a set of actions that a user can perform. Arbitrary verbs seem like they shouldn't be allowed from that definition.
No, the cluster-admin superuser role allows verbs which allows any verb


Question: importpath = "k8s.io/kubernetes/test/e2e_node/remote", deps = [ "//staging/src/k8s.io/apimachinery/pkg/util/errors:go_default_library", <ins> "//test/e2e/framework:go_default_library", </ins> "//test/e2e_node/builder:go_default_library", "//test/e2e_node/system:go_default_library", "//test/utils:go_default_library",

Which jobs are failing: Which test(s) are failing: The tests are failing in the . Since when has it been failing: 10/23/2020 Testgrid link: Reason for failure: It started failing at commit , but it's unknown if it's related to that. Anything else we need to know: We discussed in the Kubernetes SIG-Node CI subgroup and will start taking a look at it.
/assign
/sig node
Investigation notes: between successful and unsuccessful runs and found no notable differences. , and had a couple of failed health checks on initialization, which doesn't happen in the successful test runs, though there's only 2 seconds between starting readiness checks and an eventual successful check. config the same between runs looking through the kubelet logs for and , but I haven't found anything interesting yet. kubelet- failed health checks The below is a list of errors that occur in runs that succeeded and failed, which may be due to startup or specific testing scenarios. Bunch of these for numerous containers:
Both new and old test runs have a bunch of logs like this, implying that it's been having issues for a while: The failed one has this additional line: However, the doesn't have the same issues. I don't see the connection timeouts at all. I also don't see anything else in the kubelet logs that appears different between successful and failed runs. Unfortunately, I don't know enough about the infrastructure setup, but here are my current theories: about the infrastructure has gotten progressively more overloaded and now it's falling over. The tests run on 8 nodes versus 1, so they have more capacity there. Not totally buying this because why did it go from mostly successful to 100% failures rather than a mix for a bit. changed in the GCE configuration. Not sure how that is managed.
Diff there suggests three merges from k/k Only merge in test-infra is an image bump <img width="215" alt="image" src="https://user-"Interesting that a similar ci version and pull version of the job passes and the one ci version does not. pull: passing ci: failing ci: The failing ones do not have a node_e2e , but I don't see any of the expected log messages for failure of the services to start.
It is interesting that one passes and one does not.

Primary differences are the number of nodes, resource settings, and the fact that the conformance tests use Docker.
In addition to the services log missing, the overall system run logs are also missing. May be related to any decision we make for
Would we be able to kick off a Prow job with custom configuration? In other words, to rule in or rule out something in the image change, can we kick off a job with the latest everything else but use the older image? In reference to the decision on , are you suggesting we pause investigation and see if we decide to just move or change the conformance tests?
/triage accepted /priority critical-urgent
Catching up on this. qq: has anyone vetted ? Ise that pr went in around the time this job began to fail The beforesuite (AI: we should list out what it does), fails to communicate with nodes the mention of 6443 ports looks interesting here.
Good find. Definitely looks suspicious. I'll dig into that.
From : This comes from , which has some node-conformance specific test setup, so maybe something interesting there.
quick interruption for instructions on how to reproduce this one: I think this may work Focus and skip The FOCUS and SKIP arguments im getting from Image config file The job also uses the following config file Make sure to copy it and name it (the argument to IMAGECONFIG_FILE above). Additional arguments The test args to this job are I think these are the defaults already but just in case may be actually useful to add the following to the above command Miscellaneous If you run into trouble with the above commands try going through The info here came from
You'll also need to use the flag to make sure it runs in Docker, I believe.
ah yeah, good catch! Thank you. Don't know about your availability, but if you are all busy ima allocate sometime later today to dive deeper into the info we have here
I'm going to try to spend time of this in the afternoon (PDT). Specifically, I was going to try to verify that the logs from starting up the test suite look as expected and possibly try to run the case to reproduce.

My first suspicion was to have the server running in another port than 6443 (which Kubelet has configured), this is not the case: Only o.SecureServing.BindAddress is being set explicitly, but the port comes from o.NewServerRunOptions().
I think it's related to the movement of this line. I couldn't figure out how to link to the PR so I made a comment.
as you notice before there's no , and we should be seeing Before:
I can repro the issue now, and I think the line was moved upward to generate the token before starting the kubelet. Moving it up out of the if-statement may help partially, but in the conformance test the kubelet is manually started ahead of this code, so it may not matter.
Make sense. Do you mean moving this out of the conformance check?
Haha, yea, that message is deceptive... I think I'll make a PR to at least let it be more correct and give us more info.
for alerts to subscribers, PR created. Please comment.
looking green!
The output looks very strange, I have no idea why. Output has double-underscore-number for the different runs, instead of the image used. [__08] vs [cos-lts]
Ya, I noticed that. It's weird. Not sure how it's set, but you can see that the junit config names have a similar issue: -junitcos- -
Ok, found the root of the problem. Here's the command for the master suite: Note , which is not present in the conformance tests. but not sure how it eventually gets here.


Question: } else if options[0].DNS[0] != clusterNS { t.Errorf("expected nameserver %s, got %v", clusterNS, options[0].DNS[0]) } <del> if len(options[0].DNSSearch) != len(options[1].DNSSearch)+3 { </del> <ins> expLength := len(options[1].DNSSearch) + 3 if expLength > 6 { expLength = 6 } if len(options[0].DNSSearch) != expLength { </ins> t.Errorf("expected prepend of cluster domain, got %+v", options[0].DNSSearch) } else if options[0].DNSSearch[0] != ".svc."+kubelet.clusterDomain { t.Errorf("expected domain %s, got %s", ".svc."+kubelet.clusterDomain, options[0].DNSSearch)

Kubernetes version v1.3.0 Kubelet concatenates host and cluster domain entries for my pod, resulting in an invalid config (too many entries with duplicates). The max is only 6 search domains. Any more and DNS resolution breaks when trying to leverage search domains. The example below has only cluster DNS entries, but illustrates the issue: Host search nameserver 10.233.0.2 nameserver 10.90.0.1 options timeout:2 options attempts:2 Container kubectl run -i --tty busybox --image=busybox -- sh (skip waiting text) / # cat search nameserver 10.233.0.2 options attempts:2 options ndots:5 I should be enable cluster DNS config in my host DNS settings and still have functional pod DNS. Kubelet should be intelligent enough to filter out duplicate search domain entries (or drop search domains beyond the 6th entry)
I think some sane checks needs to be too. libc limits search field to 6 records and 255(?) characters. This limitation should be checked before creation.
Hey guys, I'm working on it.
Dup of


Question: const waitInterval = time.Second <ins> type metricLabelsWithValue struct { Labels []string Value  int } func TestMetrics(t *testing.T) { nonIndexedCompletion := batchv1.NonIndexedCompletion indexedCompletion := batchv1.IndexedCompletion wFinalizers := true defer featuregatetesting.SetFeatureGateDuringTest(t, feature.DefaultFeatureGate, features.JobTrackingWithFinalizers, wFinalizers)() // setup the job controller closeFn, restConfig, clientSet, ns := setup(t, "simple") defer closeFn() ctx, cancel := startJobControllerAndWaitForCaches(restConfig) defer func() { cancel() }() testCases := map[string]struct { job                            *batchv1.Job wantJobFinishedNumMetricDelta  metricLabelsWithValue wantJobPodsFinishedMetricDelta metricLabelsWithValue }{ "non-indexed job": { job: &batchv1.Job{ Spec: batchv1.JobSpec{ Completions:    pointer.Int32(2), Parallelism:    pointer.Int32(2), CompletionMode: &nonIndexedCompletion, }, }, wantJobFinishedNumMetricDelta: metricLabelsWithValue{ Labels: []string{"NonIndexed", "succeeded"}, Value:  1, }, wantJobPodsFinishedMetricDelta: metricLabelsWithValue{ Labels: []string{"NonIndexed", "succeeded"}, Value:  2, }, }, "indexed job": { job: &batchv1.Job{ Spec: batchv1.JobSpec{ Completions:    pointer.Int32(2), Parallelism:    pointer.Int32(2), CompletionMode: &indexedCompletion, }, }, wantJobFinishedNumMetricDelta: metricLabelsWithValue{ Labels: []string{"Indexed", "succeeded"}, Value:  1, }, wantJobPodsFinishedMetricDelta: metricLabelsWithValue{ Labels: []string{"Indexed", "succeeded"}, Value:  2, }, }, } job_index := 0 // job index to avoid collisions between job names created by different test cases for name, tc := range testCases { t.Run(name, func(t *testing.T) { // record the metrics after the job is created jobFinishedNumBefore, err := getCounterMetricValueForLabels(metrics.JobFinishedNum, tc.wantJobFinishedNumMetricDelta.Labels) if err != nil { t.Fatalf("Failed to collect the JobFinishedNum metric before creating the job: %q", err) } jobPodsFinishedBefore, err := getCounterMetricValueForLabels(metrics.JobPodsFinished, tc.wantJobPodsFinishedMetricDelta.Labels) if err != nil { t.Fatalf("Failed to collect the JobPodsFinished metric before creating the job: %q", err) } // create a single job and wait for its completion job := tc.job.DeepCopy() job.Name = fmt.Sprintf("job-%v", job_index) job_index++ jobObj, err := createJobWithDefaults(ctx, clientSet, ns.Name, job) if err != nil { t.Fatalf("Failed to create Job: %v", err) } validateJobPodsStatus(ctx, t, clientSet, jobObj, podsByStatus{ Active: int(*jobObj.Spec.Parallelism), Ready:  pointer.Int32(0), }, wFinalizers) if err, _ := setJobPodsPhase(ctx, clientSet, jobObj, v1.PodSucceeded, int(*jobObj.Spec.Parallelism)); err != nil { t.Fatalf("Failed setting phase %s on Job Pod: %v", v1.PodSucceeded, err) } validateJobSucceeded(ctx, t, clientSet, jobObj) // verify metric values after the job is finished validateMetricValueDeltas(t, metrics.JobFinishedNum, tc.wantJobFinishedNumMetricDelta, jobFinishedNumBefore) validateMetricValueDeltas(t, metrics.JobPodsFinished, tc.wantJobPodsFinishedMetricDelta, jobPodsFinishedBefore) }) } } func validateMetricValueDeltas(t *testing.T, counterVer *basemetrics.CounterVec, wantMetricDelta metricLabelsWithValue, metricValuesBefore metricLabelsWithValue) { t.Helper() var cmpErr error err := wait.PollImmediate(10*time.Millisecond, 10*time.Second, func() (bool, error) { cmpErr = nil metricValuesAfter, err := getCounterMetricValueForLabels(counterVer, wantMetricDelta.Labels) if err != nil { return true, fmt.Errorf("Failed to collect the %q metric after the job is finished: %q", counterVer.Name, err) } wantDelta := wantMetricDelta.Value gotDelta := metricValuesAfter.Value - metricValuesBefore.Value if wantDelta != gotDelta { cmpErr = fmt.Errorf("Unexepected metric delta for %q metric with labels %q. want: %v, got: %v", counterVer.Name, wantMetricDelta.Labels, wantDelta, gotDelta) return false, nil } return true, nil }) if err != nil { t.Errorf("Failed waiting for expected metric delta: %q", err) } if cmpErr != nil { t.Error(cmpErr) } } func getCounterMetricValueForLabels(counterVec *basemetrics.CounterVec, labels []string) (metricLabelsWithValue, error) { var result metricLabelsWithValue = metricLabelsWithValue{Labels: labels} value, err := testutil.GetCounterMetricValue(counterVec.WithLabelValues(labels...)) if err != nil { return result, err } result.Value = int(value) return result, nil } </ins> // TestJobPodFailurePolicyWithFailedPodDeletedDuringControllerRestart verifies that the job is properly marked as Failed // in a scenario when the job controller crashes between removing pod finalizers and marking the job as Failed (based on // the pod failure policy). After the finalizer for the failed pod is removed we remove the failed pod. This step is

Occasionally job completions are counted twice (or potentially multiple times?) in the metric. This happens for about 20% of jobs, but may depend on the Job specificity, which results in an overestimation of the number of finished jobs. The metric reflects the actual number of completed jobs. a kind cluster with prometheus monitoring stack (for example follow this: ). the number of completed jobs by inspecting the metric and delete a job multiple times in a loop Example job: Example repro script: . Check the number of completed jobs by inspecting the metric and the job Yes, in the kube-controller-logs we find the following log lines which are related and probably indicate repeated updated which results in double counting. <details</details<detailsReproducible with kind </details<details</details<details</details<details</details<details</details
/assign
FYI
/sig apps
/triage accepted


Question: `api_servers` | (Optional) The IP address / host name where a kubelet can get read-only access to kube-apiserver `cbr-cidr` | (Optional) The minion IP address range used for the docker container bridge. `cloud` | (Optional) Which IaaS platform is used to host kubernetes, *gce*, *azure*, *aws*, *vagrant* <del> `etcd_servers` | (Optional) Comma-delimited list of IP addresses the kube-apiserver and kubelet use to reach etcd.  Uses the IP of the first machine in the kubernetes_master role. `hostnamef` | (Optional) The full host name of the machine, i.e. hostname -f </del> <ins> `etcd_servers` | (Optional) Comma-delimited list of IP addresses the kube-apiserver and kubelet use to reach etcd.  Uses the IP of the first machine in the kubernetes_master role, or 127.0.0.1 on GCE. `hostnamef` | (Optional) The full host name of the machine, i.e. hostname -f (only used on Azure) </ins> `node_ip` | (Optional) The IP address to use to address this node `minion_ip` | (Optional) Mapped to the kubelet hostname_override, K8S TODO - change this name `network_mode` | (Optional) Networking model to use among nodes: *openvswitch*

We need to document all the configuration derived by Salt. Ideally this would be a markdown doc that we kept as live documentation within the project. The main reasons to bother: It's hard right now for anyone approaching the directory to get a big picture view of what's is going on, especially across providers. If we ever want to do something disciplined w.r.t. upgrade between (non-dev) versions, we're probably going to need a place to look for the state of the world. Decreasing the velocity with which people can add to Salt configuration will encourage people to delete more Salt configuration. cc


Question: go controller.Run(1, stopCh) } <ins> if controller, ok := s.Cert.(dynamiccertificates.ControllerRunner); ok { // runonce to be sure that we have a value. if err := controller.RunOnce(); err != nil { return nil, err } go controller.Run(1, stopCh) } </ins> // runonce to be sure that we have a value. if err := dynamicCertificateController.RunOnce(); err != nil {

When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then.

It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } if s.JSONPatchMaxCopyBytes < 0 { <del> errors = append(errors, fmt.Errorf("--json-patch-max-copy-bytes can not be negative value")) </del> <ins> errors = append(errors, fmt.Errorf("ServerRunOptions.JSONPatchMaxCopyBytes can not be negative value")) </ins> } if s.MaxRequestBodyBytes < 0 { <del> errors = append(errors, fmt.Errorf("--max-resource-write-bytes can not be negative value")) </del> <ins> errors = append(errors, fmt.Errorf("ServerRunOptions.MaxRequestBodyBytes can not be negative value")) </ins> } if err := validateHSTSDirectives(s.HSTSDirectives); err != nil {

kube-apiserver has no --max-resource-write-bytes & --json-patch-max-copy-bytes option. Correct the error message to not use the "--max-resource-write-bytes" & "--json-patch-max-copy-bytes" string. Check the source code: No response v1.19 ~ v1.23 eg: </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /area apiserver /kind cleanup /assign FYI:
/cc /triage accepted


Question: Then run `$ ./build.sh`, this will download all the needed binaries into `./binaries`. <del> You can customize your etcd version, flannel version, k8s version by changing variable `ETCD_VERSION` , `FLANNEL_VERSION` and `K8S_VERSION` in build.sh, default etcd version is 2.0.9, flannel version is 0.4.0 and K8s version is 0.15.0. </del> <ins> You can customize your etcd version, flannel version, k8s version by changing variable `ETCD_VERSION` , `FLANNEL_VERSION` and `K8S_VERSION` in build.sh, default etcd version is 2.0.9, flannel version is 0.4.0 and K8s version is 0.18.0. </ins> Please make sure that there are `kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `kubelet`, `kube-proxy`, `etcd`, `etcdctl` and `flannel` in the binaries/master or binaries/minion directory.

This flag is deprecated and no longer has any effect.
I wonder which version does this include.Now ubuntu approach is using ubuntu 0.15.0.
This change was checked in this week, so it will be true for clusters at 0.18.0+ (once the next release is cut next week). So the next time the ubuntu documentation is updated, this change should be reflected.
Ohit means that minion node can be automatically discovered by the master node when then kubelet set ? No other configuration is needed?
Yes, the kubelet, when passed the location of the apiserver, will register itself and then post node update messages. The only part that may depend on your deployment environment is the set of addresses that the kubelet passes to the apiserver. They will be based on the name of the node, reverse resolving the name of the node, or checking for addresses from the cloud provider.
Got it.I will include this when I updates the doc and scripts. Besides, I have a little questions here. Eric told me the addon services do not have service-account now.Will 0.18.0+ have it?
Service accounts will be included as part of 0.18.0 as well.
got it
- makes sense


Question: clock:        c, duration:     duration, jitter:       jitter, <del> backoffTimer: c.NewTimer(0), </del> <ins> backoffTimer: nil, </ins> } }

I noticed that this code: was printing which should not happen. Also occurs in wait.Forever. I suspect that in regressed this as that was the most recent major update and I only started seeing this on a recent apimachinery bump.
/sig api-machinery /assign as the reviewer
GitHub didn't allow me to assign the following users: zhan849. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig api-machinery /assign as the reviewer Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
This is a correctness regression because the wait methods expect not to be run run more often than the period and may guard expensive behavior. Suggest a revert followed by a more rigorous test and review on merge.
thanks for reporting this, and I apologize about the regression I made. I did some quick test, and I am able to reproduce it occasionally, which indicate a race somewhere: Test: Failed output: Successful output? In the original code I checked in, I initialized the timer in the backoff manager construction time: However, after I changed to initializing the timer when is called, I'm not able to reproduce the race. The results indicated that the timer initialization is likely to be the place that caused the regression , and seems to be able to get fixed by change the way the timer gets initialized. However I'm unable to identify why it could cause such race the first time. Are you able to provide any insights? /cc
why are we initializing a timer with a zero duration? that looks related to this issue:
I see us resetting a timer we have no guarantee is stopped/expired:
dropped the information needed to accurately stop/reset the timer as documented
Hi thanks for providing the information! seems like the correct place to initialize the timer is in the "Backoff" method, since zero duration is problematic.

When I was putting up the interface, I documented that the timer will not be drained, and the reason is that returns a timer explicitly for user to block on that, so it might be confusing to block in itself Shall we change the impl to always drain the channel in backoff instead or relying on user to use it correctly? with this being said, will be changed to , which means that the will still rely on user's providing information about whether the channel is drained or not.
/cc thoughts? ^^
- sorry for that - we hit the bug with resetting timer not the first time, I should have caught that (I remember at least two other bugs with that). I'm not sure revert is a good option, because there were PRs that were chained on that: - this would have to be a very careful manual revert which I'm not sure it justifies the risk, given that 1.18 is already out. Currently the behavior is unpredictable, because if getNextbackoof() called for the first time is non-zero, it may or may not be ready immediately depending on the race. That is obviously bad. Initializing with infinity may solve this particular problem, but I need to think more about it to ensure that we don't have any problems in the later phase too...
thanks ! and sry again I did not read the timer's doc carefully enough but simply followed the Clock interface :( To fix, we can also initialize the timer to nil and create a fresh one when is first called, it's safe as this component is designed to run in single-threaded environment. But to ensure timer is drained before reset, we need to have user explicitly pass in "if timer is drained" into the to accurately decide if needs to drain. Or, is it ok to document the behavior and assume user will drain on before calling next ?
Very quick (I might have missed something) glance from the code suggests, that the only place we would need to change is actually BackoffUntil method. So passing the "if timer is drained" bit seems less error prone imho.
We should avoid changing the interface in a patch release.

I think initializing the timer in Backoff() and documenting the returned timer must be expired and drained before calling Backoff() again is the most compatible thing we can do:
Thanks and I'm also more inclined to Jordan's proposal, besides avoid interface change, if user are able to accurately pass "if timer is drained" info, it's fair to let user finish draining. do you have a PR already? of if not, I can make one, and add more tests if possible about this failure case.
I do not have a PR open


Question: createReplicaSetOrDie(cs, ns, 3, false) ginkgo.By("First trying to evict a pod which shouldn't be evictable") <ins> waitForPodsOrDie(cs, ns, 3) // make sure that they are running and so would be evictable with a different pdb </ins> pod, err := locateRunningPod(cs, ns) framework.ExpectNoError(err) <del> waitForPodsOrDie(cs, ns, 3) // make sure that they are running and so would be evictable with a different pdb </del> e := &policyv1beta1.Eviction{ ObjectMeta: metav1.ObjectMeta{ Name:      pod.Name,

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: all of the e2e jobs that run this test Which test(s) are flaking: ` Testgrid link: Reason for failure: mostly: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated /sig apps xref:
approx:
my suspicion: /
I can't find anything else that seems relevant in the time frame, and it is the only change to the test file in that time frame.
merged recently, not sure if it's relevant
Not sure what the procedure it, but we can revert or try some other image instead of Pause, if you think it helps. I don't have a good explanation why the change would make the test flake like this.
I don't have a good explanation, just that it affects the test and is in the time frame that seems to fit the flake spike It may take a bit to pin down what exactly happened here :/ I can't look for a little bit.
I tried to reproduce this behaviour on a test cluster in AWS, but couldn't. I prepared a PR to revert the change, just incase.
Good news and bad news kind of thing... The same test fails in the revert PR also, with the previous EchoServer image:
I did an experiment reverting the second suspect in I couldn't reproduce the issue anymore even though I tried. You can check the results in the PR history.
I believe I have found the issue here. More information and fix incoming
Thanks for the update
I just about have a fix ready here. I was able to narrow the commit range to by looking at when this flake first occurred in It looks like was correct that this was introduced in . I believe what is happening is that in the test we are getting getting a with a set (the one that we previously successfully issued an eviction for) before waiting for 3 pods to be ready again in the namespace (). Before the aforementioned PR caused us to skip checking PDBs when is set we would still report back that eviction was not possible because our with did not meet the other criteria (i.e.

was not , , or ), which caused us to still check PDBs, which would show us that we did not have the budget to delete (even though it was not really taking into account the that we were actually attempting to evict). I think there are a few things we can do to address this: we should check that not only is the , but also that it does not have set. above where we call so that we do not attempt to get a before all of the pods are ready. in that the does not have a . Just doing (3) by itself would address our current problem, but (1) and (2) also seem to make sense in the context of what we are trying to check here.
Thanks everyone and apologies for the false alarm :-)
/lifecycle active /priority critical-urgent (this is blocking merges of other fixes) /assign
/milestone v1.19


Question: var ( service1 = "svc1" service2 = "svc2" <ins> service3 = "svc.foo" service4 = "svc_foo" service5 = "svc@foo" service6 = "svc:foo" </ins> invalid1 = "svcn" invalid2 = "svc.foon" )

I enabled the on EKS 1.27.4 using Ubuntu 20.04.6. When I try to query the kubelet logs or log files from the system I get the error "input contains unsupported characters" The systemd kubelet service runs as a snap called and I think that might be the source of this problem (running the kubelet in a snap), but I'm not sure if it's intended to work. The fact that I couldn't see /var/log file query was surprising and the error message is misleading which is why I'm opening an issue here. I followed the same configuration steps on Amazon Linux 2 and it works as expected. Log query should work on Ubuntu or provide a clearer error message. This should be reproducible on any Kubernetes cluster running Ubuntu with Kubelet in a snap, but this is what I did. Install to make cluster creation and karpenter setup easier) Install Karpenter as a node autoscaler. This will automatically create a provisioner and AWSNodeTemplate objects. Modify the AWSNodeTemplate with userdata that can customize Ubuntu with log query enabled Deploy a workload that uses hostname spread to get more nodes. When the new Ubuntu node is created you can log into the node with session manager. I like to use You can verify that the kubelet is running and logging is enabled Now you can try to query logs This works with AL2. You can test with a different AWSNodeTemplate that uses AL2 instead <details</details<detailsEKS (but should happen on any provider) </details<details</details<details</details<detailscontainerd </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
it's always best to talk to the product support before opening the upstream ticket (this repository in this case). the k8s docs do not have a guide for running the kubelet as a snap. it could be an entirely snap related bug, indeed. please communicate with EKS and the owners of the snap package.

if they confirm a kubernetes/kubernetes bug, you can reopen this ticket. thanks
/sig node /triage needs-information /close to discuss with other users:
Closing this issue. <detailsIn response to : /sig node /triage needs-information /close </details
After digging into the error a bit more I found the problem was actually the regex match for the service name. I opened to fix it.


Question: return createdService, httpPort } <ins> // testEndpointReachability tests reachability to endpoints (i.e. IP, ServiceName) and ports. Test request is initiated from specified execPod. // TCP and UDP protocol based service are supported at this moment func testEndpointReachability(ctx context.Context, endpoint string, port int32, protocol v1.Protocol, execPod *v1.Pod, timeout time.Duration) error { cmd := "" switch protocol { case v1.ProtocolTCP: cmd = fmt.Sprintf("echo hostName | nc -v -t -w 2 %s %v", endpoint, port) case v1.ProtocolUDP: cmd = fmt.Sprintf("echo hostName | nc -v -u -w 2 %s %v", endpoint, port) default: return fmt.Errorf("service reachability check is not supported for %v", protocol) } err := wait.PollImmediateWithContext(ctx, framework.Poll, timeout, func(ctx context.Context) (bool, error) { stdout, err := e2eoutput.RunHostCmd(execPod.Namespace, execPod.Name, cmd) if err != nil { framework.Logf("Service reachability failing with error: %vnRetrying...", err) return false, nil } trimmed := strings.TrimSpace(stdout) if trimmed != "" { return true, nil } return false, nil }) if err != nil { return fmt.Errorf("service is not reachable within %v timeout on endpoint %s %d over %s protocol", timeout, endpoint, port, protocol) } return nil } </ins>

see A pod listens on port N, on both TCP and UDP. A service forwards to port N, only TCP. Accessing the service via UDP should fail.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
/assign this was caught with hostPort , but we should have one for Services